[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:05 dist-test-slave-lgr5 krb5kdc[3900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:05 dist-test-slave-lgr5 krb5kdc[3900](info): set up 2 sockets
Aug 11 12:33:05 dist-test-slave-lgr5 krb5kdc[3900](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 12:33:06.609238 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:06 dist-test-slave-lgr5 krb5kdc[3900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:06.802851 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:45067
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:45067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:07.265374 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:07.265795 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:07.265895 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:07.274381 3935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:07.274586 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:07.274689 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:07.274809 3935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:07.274865 3935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:07.286785 3935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:45067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:45067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:07.288892 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:07.290886 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:07.310364 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:07.310704 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:07.310784 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:07.312436 3935 server_base.cc:1034] running on GCE node
I20260811 12:33:07.313710 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:07.316267 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:07.317543 3935 hybrid_clock.cc:648] HybridClock initialized: now 1786451587317448 us; error 87 us; skew 500 ppm
I20260811 12:33:07.318142 3935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:07 dist-test-slave-lgr5 krb5kdc[3900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:07.322074 3935 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:07.324823 3935 webserver.cc:460] Webserver started at http://127.3.204.126:33415/ using document root <none> and password file <none>
I20260811 12:33:07.326045 3935 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:07.326246 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:07.326787 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:07.331405 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "6dba84486f474308a6723f7734c8c228"
format_stamp: "Formatted at 2026-08-11 12:33:07 on dist-test-slave-lgr5"
server_key: "df55673b2d953142ea098a3bd9ba81fd"
server_key_iv: "3f9ec1f64ae92018c862c758280b6656"
server_key_version: "encryptionkey@0"
I20260811 12:33:07.332475 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "6dba84486f474308a6723f7734c8c228"
format_stamp: "Formatted at 2026-08-11 12:33:07 on dist-test-slave-lgr5"
server_key: "df55673b2d953142ea098a3bd9ba81fd"
server_key_iv: "3f9ec1f64ae92018c862c758280b6656"
server_key_version: "encryptionkey@0"
I20260811 12:33:07.341173 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260811 12:33:07.347528 3959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:07.349922 3935 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260811 12:33:07.350246 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "6dba84486f474308a6723f7734c8c228"
format_stamp: "Formatted at 2026-08-11 12:33:07 on dist-test-slave-lgr5"
server_key: "df55673b2d953142ea098a3bd9ba81fd"
server_key_iv: "3f9ec1f64ae92018c862c758280b6656"
server_key_version: "encryptionkey@0"
I20260811 12:33:07.350466 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:07.370486 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:07.375880 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:07.376242 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:07.408880 4048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:45067 every 8 connection(s)
I20260811 12:33:07.408985 3935 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:45067
I20260811 12:33:07.411470 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:07.416203 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 3935
I20260811 12:33:07.416687 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:07.419792 3889 external_mini_cluster.cc:1428] Setting key f57f4d1107bf1b68c023a011f390abd7
I20260811 12:33:07.420156 4049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:07.438969 4049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228: Bootstrap starting.
I20260811 12:33:07.446501 4049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:07.448493 4049 log.cc:824] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:07.453681 4049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228: No bootstrap required, opened a new log
Aug 11 12:33:07 dist-test-slave-lgr5 krb5kdc[3900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:07.465557 4049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dba84486f474308a6723f7734c8c228" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45067 } }
I20260811 12:33:07.466310 4049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:07.466529 4049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dba84486f474308a6723f7734c8c228, State: Initialized, Role: FOLLOWER
I20260811 12:33:07.467828 4049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dba84486f474308a6723f7734c8c228" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45067 } }
I20260811 12:33:07.468218 4049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:07.468500 4049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:07.468714 4049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:07.471485 4049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dba84486f474308a6723f7734c8c228" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45067 } }
I20260811 12:33:07.473065 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:07.427441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.3.204.126:45067)
0811 12:33:07.428796 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:07.428816 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:07.432525 (+ 3709us) server_negotiation.cc:407] Connection header received
0811 12:33:07.444435 (+ 11910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:07.444470 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:07.445233 (+ 763us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:07.445823 (+ 590us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:07.449200 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:07.451576 (+ 2376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:07.453593 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:07.454924 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:07.455238 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:07.462247 (+ 7009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:07.462318 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:07.462358 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:07.462406 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:07.466167 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:07.467222 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:07.467230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:07.467236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:07.467555 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:07.468061 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:07.468065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:07.468067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:07.469056 (+ 989us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:07.469550 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:07.470513 (+ 963us) server_negotiation.cc:299] Negotiation successful
0811 12:33:07.470942 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":541,"threads_started":1}
I20260811 12:33:07.472973 4049 leader_election.cc:302] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dba84486f474308a6723f7734c8c228; no voters:
I20260811 12:33:07.474014 4049 leader_election.cc:288] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:07.474220 4060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:07.475633 4060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [term 1 LEADER]: Becoming Leader. State: Replica: 6dba84486f474308a6723f7734c8c228, State: Running, Role: LEADER
I20260811 12:33:07.476852 4060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dba84486f474308a6723f7734c8c228" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45067 } }
I20260811 12:33:07.477375 4049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:07.483713 4062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dba84486f474308a6723f7734c8c228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dba84486f474308a6723f7734c8c228" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45067 } } }
I20260811 12:33:07.483701 4063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dba84486f474308a6723f7734c8c228. Latest consensus state: current_term: 1 leader_uuid: "6dba84486f474308a6723f7734c8c228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dba84486f474308a6723f7734c8c228" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45067 } } }
I20260811 12:33:07.484282 4063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:07.484282 4062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:07.489825 4078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:07.501227 4078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:07.516927 4078 catalog_manager.cc:1347] Generated new cluster ID: 92b95722dd824a11b1b0c5a3ac21c6e5
I20260811 12:33:07.517055 4078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:07.535182 4078 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:07.537427 4078 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:07.551932 4078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dba84486f474308a6723f7734c8c228: Generated new TSK 0
I20260811 12:33:07.553372 4078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:07.601426 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45067
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:08.028395 4097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:08.028839 4097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:08.029040 4097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:08.038424 4097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:08.038551 4097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:08.038717 4097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:08.050328 4097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:08.051961 4097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:08.053815 4097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:08.070013 4111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:08.070013 4110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:08.070008 4113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:08.070667 4097 server_base.cc:1034] running on GCE node
I20260811 12:33:08.071383 4097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:08.072428 4097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:08.073652 4097 hybrid_clock.cc:648] HybridClock initialized: now 1786451588073596 us; error 47 us; skew 500 ppm
I20260811 12:33:08.074000 4097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:08 dist-test-slave-lgr5 krb5kdc[3900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:08.077842 4097 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:08.079674 4097 webserver.cc:460] Webserver started at http://127.3.204.65:34531/ using document root <none> and password file <none>
I20260811 12:33:08.080480 4097 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:08.080721 4097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:08.081102 4097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:08.084627 4097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "608d5846e8a0464aa369e725c53884cd"
format_stamp: "Formatted at 2026-08-11 12:33:08 on dist-test-slave-lgr5"
server_key: "d3d3ea90de33740674dae81670802079"
server_key_iv: "305cf2cd5bff97af82c8d96db3f61b57"
server_key_version: "encryptionkey@0"
I20260811 12:33:08.085403 4097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "608d5846e8a0464aa369e725c53884cd"
format_stamp: "Formatted at 2026-08-11 12:33:08 on dist-test-slave-lgr5"
server_key: "d3d3ea90de33740674dae81670802079"
server_key_iv: "305cf2cd5bff97af82c8d96db3f61b57"
server_key_version: "encryptionkey@0"
I20260811 12:33:08.090978 4097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:08.094900 4122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:08.096529 4097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 12:33:08.096709 4097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "608d5846e8a0464aa369e725c53884cd"
format_stamp: "Formatted at 2026-08-11 12:33:08 on dist-test-slave-lgr5"
server_key: "d3d3ea90de33740674dae81670802079"
server_key_iv: "305cf2cd5bff97af82c8d96db3f61b57"
server_key_version: "encryptionkey@0"
I20260811 12:33:08.096930 4097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:08.119372 4097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:08.124363 4097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:08.124651 4097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:08.125852 4097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:08.128103 4097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:08.128208 4097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 12:33:08.128417 4097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:08.128489 4097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:08.177692 4097 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:35903
I20260811 12:33:08.177692 4320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:35903 every 8 connection(s)
I20260811 12:33:08.179374 4097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:08.183063 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 4097
I20260811 12:33:08.183492 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:08.184818 3889 external_mini_cluster.cc:1428] Setting key f9f9c0baf4195e2c5ef0c23c5aaa0a53
Aug 11 12:33:08 dist-test-slave-lgr5 krb5kdc[3900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
I20260811 12:33:08.205308 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:08.183032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:48843 (local address 127.3.204.126:45067)
0811 12:33:08.185190 (+ 2158us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:08.185196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:08.185221 (+ 25us) server_negotiation.cc:407] Connection header received
0811 12:33:08.187329 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:08.187335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:08.187420 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:08.187516 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:08.189766 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:08.190844 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:08.192630 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:08.193275 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:08.193409 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:08.197646 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:08.197656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:08.197661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:08.197703 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:08.201534 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:08.202497 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:08.202503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:08.202507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:08.202692 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:08.203234 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:08.203240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:08.203243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:08.203648 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:08.203819 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:08.204777 (+ 958us) server_negotiation.cc:299] Negotiation successful
0811 12:33:08.204995 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2025,"thread_start_us":1914,"threads_started":1}
I20260811 12:33:08.206496 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:08.183512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45067 (local address 127.3.204.65:48843)
0811 12:33:08.184447 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:08.184497 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:08.187009 (+ 2512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:08.187859 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:08.187868 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:08.188795 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:08.189577 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:08.189599 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:08.191045 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:08.191049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:08.192480 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:08.192494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:08.193547 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:08.193555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:08.193873 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:08.194631 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:08.194666 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:08.197341 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:08.201727 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:08.201738 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:08.201753 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:08.202345 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:08.202869 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:08.202873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:08.202876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:08.203094 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:08.203838 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:08.203845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:08.204280 (+ 435us) client_negotiation.cc:769] Sending connection context
0811 12:33:08.204646 (+ 366us) client_negotiation.cc:240] Negotiation successful
0811 12:33:08.205090 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":338,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
I20260811 12:33:08.208500 4321 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45067
I20260811 12:33:08.208969 4321 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:08.209923 4321 heartbeater.cc:511] Master 127.3.204.126:45067 requested a full tablet report, sending...
I20260811 12:33:08.214308 3984 ts_manager.cc:194] Registered new tserver with Master: 608d5846e8a0464aa369e725c53884cd (127.3.204.65:35903)
I20260811 12:33:08.216609 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:48843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:08.232718 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45067
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:08.657248 4328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:08.657660 4328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:08.657843 4328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:08.666042 4328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:08.666175 4328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:08.666345 4328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:08.678666 4328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:08.680107 4328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:08.681610 4328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:08.697973 4335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:08.698087 4338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:08.697906 4340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:08.699373 4328 server_base.cc:1034] running on GCE node
I20260811 12:33:08.700121 4328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:08.701201 4328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:08.702423 4328 hybrid_clock.cc:648] HybridClock initialized: now 1786451588702367 us; error 51 us; skew 500 ppm
I20260811 12:33:08.702947 4328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:08 dist-test-slave-lgr5 krb5kdc[3900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:08.707487 4328 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:08.709297 4328 webserver.cc:460] Webserver started at http://127.3.204.66:44569/ using document root <none> and password file <none>
I20260811 12:33:08.710045 4328 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:08.710175 4328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:08.710490 4328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:08.713614 4328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "5822a7bdedeb4690af1fb3a051feb864"
format_stamp: "Formatted at 2026-08-11 12:33:08 on dist-test-slave-lgr5"
server_key: "bf5abb262ab821cf86054bf5b76ffd9f"
server_key_iv: "e84e38818d041089bae42f880ccd8ae7"
server_key_version: "encryptionkey@0"
I20260811 12:33:08.714398 4328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "5822a7bdedeb4690af1fb3a051feb864"
format_stamp: "Formatted at 2026-08-11 12:33:08 on dist-test-slave-lgr5"
server_key: "bf5abb262ab821cf86054bf5b76ffd9f"
server_key_iv: "e84e38818d041089bae42f880ccd8ae7"
server_key_version: "encryptionkey@0"
I20260811 12:33:08.720427 4328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 12:33:08.724898 4355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:08.726437 4328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:33:08.726606 4328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "5822a7bdedeb4690af1fb3a051feb864"
format_stamp: "Formatted at 2026-08-11 12:33:08 on dist-test-slave-lgr5"
server_key: "bf5abb262ab821cf86054bf5b76ffd9f"
server_key_iv: "e84e38818d041089bae42f880ccd8ae7"
server_key_version: "encryptionkey@0"
I20260811 12:33:08.726812 4328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:08.759045 4328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:08.764493 4328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:08.764962 4328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:08.766165 4328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:08.768040 4328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:08.768177 4328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 12:33:08.768307 4328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:08.768371 4328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:08.816387 4328 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:45047
I20260811 12:33:08.816461 4575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:45047 every 8 connection(s)
I20260811 12:33:08.817823 4328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:08.824363 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 4328
I20260811 12:33:08.824545 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:08.824975 3889 external_mini_cluster.cc:1428] Setting key 9570910c00920be5ac2f61df9d45d7b5
Aug 11 12:33:08 dist-test-slave-lgr5 krb5kdc[3900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:08.843047 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:08.820999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:45157 (local address 127.3.204.126:45067)
0811 12:33:08.821687 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:08.821691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:08.822985 (+ 1294us) server_negotiation.cc:407] Connection header received
0811 12:33:08.825149 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:08.825155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:08.825241 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:08.825381 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:08.827623 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:08.828856 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:08.831090 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:08.831844 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:08.831988 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:08.835695 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:08.835702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:08.835708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:08.835752 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:08.839752 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:08.840614 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:08.840619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:08.840622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:08.840761 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:08.841225 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:08.841229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:08.841231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:08.841540 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:08.841678 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:08.842450 (+ 772us) server_negotiation.cc:299] Negotiation successful
0811 12:33:08.842688 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":459,"threads_started":1}
I20260811 12:33:08.844117 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:08.821821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45067 (local address 127.3.204.66:45157)
0811 12:33:08.822809 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:08.822862 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:08.824859 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:08.825769 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:08.825781 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:08.826515 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:08.827426 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:08.827449 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:08.829099 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:08.829103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:08.830932 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:08.830946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:08.832118 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:08.832126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:08.832369 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:08.833059 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:08.833082 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:08.835420 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:08.839934 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:08.839943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:08.839959 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:08.840484 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:08.840879 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:08.840883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:08.840885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:08.841105 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:08.841671 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:08.841679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:08.842067 (+ 388us) client_negotiation.cc:769] Sending connection context
0811 12:33:08.842321 (+ 254us) client_negotiation.cc:240] Negotiation successful
0811 12:33:08.842906 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":522,"threads_started":1}
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
I20260811 12:33:08.845697 4576 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45067
I20260811 12:33:08.846094 4576 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:08.847122 4576 heartbeater.cc:511] Master 127.3.204.126:45067 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
I20260811 12:33:08.848946 3984 ts_manager.cc:194] Registered new tserver with Master: 5822a7bdedeb4690af1fb3a051feb864 (127.3.204.66:45047)
I20260811 12:33:08.850193 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:45157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:08.872344 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45067
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:33:09.221475 4321 heartbeater.cc:503] Master 127.3.204.126:45067 was elected leader, sending a full tablet report...
W20260811 12:33:09.295320 4592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:09.295691 4592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:09.295820 4592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:09.304248 4592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:09.304366 4592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:09.304486 4592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:09.315064 4592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:09.316432 4592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:09.317840 4592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:09.333657 4610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:09.333696 4604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:09.334331 4603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:09.335048 4592 server_base.cc:1034] running on GCE node
I20260811 12:33:09.335693 4592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:09.336872 4592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:09.338114 4592 hybrid_clock.cc:648] HybridClock initialized: now 1786451589338048 us; error 60 us; skew 500 ppm
I20260811 12:33:09.338472 4592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:09 dist-test-slave-lgr5 krb5kdc[3900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451589, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:09.342008 4592 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:09.343799 4592 webserver.cc:460] Webserver started at http://127.3.204.67:41271/ using document root <none> and password file <none>
I20260811 12:33:09.344579 4592 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:09.344725 4592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:09.345084 4592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:09.348538 4592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "6e96ed6e77874fe4a916164bdcef9f6c"
format_stamp: "Formatted at 2026-08-11 12:33:09 on dist-test-slave-lgr5"
server_key: "f6302c475c4d4d0f35f9d88aac3f161c"
server_key_iv: "df4f9f8c0659826c5cbd4ad0e2318ae6"
server_key_version: "encryptionkey@0"
I20260811 12:33:09.349383 4592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "6e96ed6e77874fe4a916164bdcef9f6c"
format_stamp: "Formatted at 2026-08-11 12:33:09 on dist-test-slave-lgr5"
server_key: "f6302c475c4d4d0f35f9d88aac3f161c"
server_key_iv: "df4f9f8c0659826c5cbd4ad0e2318ae6"
server_key_version: "encryptionkey@0"
I20260811 12:33:09.355726 4592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260811 12:33:09.360206 4632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:09.361819 4592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:09.361979 4592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "6e96ed6e77874fe4a916164bdcef9f6c"
format_stamp: "Formatted at 2026-08-11 12:33:09 on dist-test-slave-lgr5"
server_key: "f6302c475c4d4d0f35f9d88aac3f161c"
server_key_iv: "df4f9f8c0659826c5cbd4ad0e2318ae6"
server_key_version: "encryptionkey@0"
I20260811 12:33:09.362171 4592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:09.379998 4592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:09.385135 4592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:09.385617 4592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:09.387255 4592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:09.389135 4592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:09.389217 4592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:09.389326 4592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:09.389370 4592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:09.439898 4592 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:34643
I20260811 12:33:09.440133 4836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:34643 every 8 connection(s)
I20260811 12:33:09.441278 4592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:09.443501 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 4592
I20260811 12:33:09.443778 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:09.444474 3889 external_mini_cluster.cc:1428] Setting key dc1a066d766767251fd3f2a086153c36
Aug 11 12:33:09 dist-test-slave-lgr5 krb5kdc[3900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451589, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:09.463002 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:09.444270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44125 (local address 127.3.204.126:45067)
0811 12:33:09.444788 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:09.444795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:09.446208 (+ 1413us) server_negotiation.cc:407] Connection header received
0811 12:33:09.448367 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:09.448373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:09.448445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:09.448547 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:09.450378 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.451146 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.452745 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.453150 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.453259 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:09.456942 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:09.456947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:09.456950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:09.456983 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:09.459766 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.460741 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.460745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.460747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.460932 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.461409 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.461414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.461416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.461775 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:09.461902 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:09.462541 (+ 639us) server_negotiation.cc:299] Negotiation successful
0811 12:33:09.462709 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260811 12:33:09.463914 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:09.445084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45067 (local address 127.3.204.67:44125)
0811 12:33:09.446050 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:09.446100 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:09.448108 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:09.448866 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:09.448876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:09.449628 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:09.450211 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:09.450230 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.451300 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.451304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:09.452607 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:09.452621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.453369 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.453377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:09.453598 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:09.454185 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:09.454207 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:09.456732 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:09.459940 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:09.459949 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:09.459968 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:09.460612 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:09.461046 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:09.461051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:09.461054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:09.461285 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:09.461889 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:09.461897 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:09.462228 (+ 331us) client_negotiation.cc:769] Sending connection context
0811 12:33:09.462458 (+ 230us) client_negotiation.cc:240] Negotiation successful
0811 12:33:09.462844 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":448,"threads_started":1}
I20260811 12:33:09.465260 4838 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45067
I20260811 12:33:09.465590 4838 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:09.466657 4838 heartbeater.cc:511] Master 127.3.204.126:45067 requested a full tablet report, sending...
I20260811 12:33:09.468124 3984 ts_manager.cc:194] Registered new tserver with Master: 6e96ed6e77874fe4a916164bdcef9f6c (127.3.204.67:34643)
I20260811 12:33:09.468907 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:44125
I20260811 12:33:09.474174 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 12:33:09 dist-test-slave-lgr5 krb5kdc[3900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 12:33:09 dist-test-slave-lgr5 krb5kdc[3900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:09.525173 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:09.506792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.3.204.65:35903)
0811 12:33:09.508109 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:09.508121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:09.508668 (+ 547us) server_negotiation.cc:407] Connection header received
0811 12:33:09.508849 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:09.508857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:09.509127 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:09.509283 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:09.511041 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.512466 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.513825 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.514418 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.514533 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:09.519116 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:09.519128 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:09.519145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:09.519181 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:09.522049 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.522711 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.522718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.522720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.522859 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.523218 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.523222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.523224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.523663 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:09.524054 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:09.524332 (+ 278us) server_negotiation.cc:299] Negotiation successful
0811 12:33:09.524648 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043,"thread_start_us":813,"threads_started":1}
W20260811 12:33:09.526599 4169 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60550
Aug 11 12:33:09 dist-test-slave-lgr5 krb5kdc[3900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 12:33:09 dist-test-slave-lgr5 krb5kdc[3900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:09.573416 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:09.558339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.3.204.65:35903)
0811 12:33:09.558588 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:09.558596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:09.558788 (+ 192us) server_negotiation.cc:407] Connection header received
0811 12:33:09.558984 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:09.558988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:09.559091 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:09.559186 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:09.560809 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.561637 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.563140 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.563584 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.563684 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:09.567889 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:09.567894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:09.567898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:09.567939 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:09.570657 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.571476 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.571481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.571483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.571633 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.571969 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.571974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.571976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.572337 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:09.572537 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:09.572859 (+ 322us) server_negotiation.cc:299] Negotiation successful
0811 12:33:09.573050 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 12:33:09.575655 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 4097
I20260811 12:33:09.590051 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:09.576686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.3.204.65:35903)
0811 12:33:09.576848 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:09.576854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:09.577358 (+ 504us) server_negotiation.cc:407] Connection header received
0811 12:33:09.577531 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:09.577534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:09.577594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:09.577670 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:09.579258 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.579972 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.581616 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.582070 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.582184 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:09.584696 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:09.584700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:09.584703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:09.584739 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:09.587305 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.588001 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.588006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.588008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.588217 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.588602 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.588607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.588611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.588900 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:09.589124 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:09.589445 (+ 321us) server_negotiation.cc:299] Negotiation successful
0811 12:33:09.589671 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 12:33:09.590965 4310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:09.701566 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 4097
I20260811 12:33:09.723940 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 4328
Aug 11 12:33:09 dist-test-slave-lgr5 krb5kdc[3900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:33:09.742688 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:09.724976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.3.204.66:45047)
0811 12:33:09.725598 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:09.725605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:09.725631 (+ 26us) server_negotiation.cc:407] Connection header received
0811 12:33:09.725697 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:09.725703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:09.725913 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:09.726059 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:09.727416 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.728785 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.730138 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.730744 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.730880 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:09.735774 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:09.735791 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:09.735808 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:09.735851 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:09.739103 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.739919 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.739928 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.739931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.740153 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.740723 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.740728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.740730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.741145 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:09.741384 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:09.742004 (+ 620us) server_negotiation.cc:299] Negotiation successful
0811 12:33:09.742315 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":328,"threads_started":1}
I20260811 12:33:09.744043 4566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:09.828991 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 4328
I20260811 12:33:09.849524 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 4592
Aug 11 12:33:09 dist-test-slave-lgr5 krb5kdc[3900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:33:09.866887 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:09.850703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.3.204.67:34643)
0811 12:33:09.851391 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:09.851399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:09.851431 (+ 32us) server_negotiation.cc:407] Connection header received
0811 12:33:09.851498 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:09.851504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:09.851724 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:09.852031 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:09.853371 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.854502 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.855798 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:09.856466 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:09.856588 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:09.860986 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:09.860999 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:09.861016 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:09.861057 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:09.863817 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.864651 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.864659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.864661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.864799 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:09.865208 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:09.865212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:09.865214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:09.865614 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:09.865906 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:09.866255 (+ 349us) server_negotiation.cc:299] Negotiation successful
0811 12:33:09.866527 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":352,"threads_started":1}
I20260811 12:33:09.868758 4830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:09.951875 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 4592
I20260811 12:33:09.973069 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 3935
I20260811 12:33:09.974433 4040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:10.078711 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 3935
2026-08-11T12:33:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4743 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:10 dist-test-slave-lgr5 krb5kdc[4891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:10 dist-test-slave-lgr5 krb5kdc[4891](info): set up 2 sockets
Aug 11 12:33:10 dist-test-slave-lgr5 krb5kdc[4891](info): commencing operation
krb5kdc: starting...
W20260811 12:33:11.347256 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260811 12:33:11.610615 3889 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.263s user 0.000s sys 0.006s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260811 12:33:11.871861 3889 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.261s user 0.000s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260811 12:33:12.012548 3889 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.140s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/test-user.keytab.
W20260811 12:33:12.661808 3889 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.649s user 0.000s sys 0.005s
Aug 11 12:33:12 dist-test-slave-lgr5 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:12.752130 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:43615
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:43615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:13.080951 4922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:13.081293 4922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:13.081446 4922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:13.088937 4922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:13.089077 4922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:13.089150 4922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:13.089226 4922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:13.089291 4922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:13.099490 4922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:43615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:13.101007 4922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:13.102382 4922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:13.114344 4932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:13.114509 4939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:13.114509 4936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:13.115568 4922 server_base.cc:1034] running on GCE node
I20260811 12:33:13.116441 4922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:13.118014 4922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:13.119216 4922 hybrid_clock.cc:648] HybridClock initialized: now 1786451593119172 us; error 35 us; skew 500 ppm
I20260811 12:33:13.119560 4922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:13 dist-test-slave-lgr5 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:13.132273 4922 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:13.133989 4922 webserver.cc:460] Webserver started at http://127.3.204.126:33707/ using document root <none> and password file <none>
I20260811 12:33:13.134752 4922 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:13.134891 4922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:13.135236 4922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:13.138516 4922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "5ea50db32da644a0907ea72bf4abad48"
format_stamp: "Formatted at 2026-08-11 12:33:13 on dist-test-slave-lgr5"
server_key: "7fd98e5b65dffee5aec4a6e9cce0020b"
server_key_iv: "947fdeb77fbc7edbee3dba8a2c7c9752"
server_key_version: "encryptionkey@0"
I20260811 12:33:13.139421 4922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "5ea50db32da644a0907ea72bf4abad48"
format_stamp: "Formatted at 2026-08-11 12:33:13 on dist-test-slave-lgr5"
server_key: "7fd98e5b65dffee5aec4a6e9cce0020b"
server_key_iv: "947fdeb77fbc7edbee3dba8a2c7c9752"
server_key_version: "encryptionkey@0"
I20260811 12:33:13.144968 4922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260811 12:33:13.148614 4950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:13.149995 4922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 12:33:13.150184 4922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "5ea50db32da644a0907ea72bf4abad48"
format_stamp: "Formatted at 2026-08-11 12:33:13 on dist-test-slave-lgr5"
server_key: "7fd98e5b65dffee5aec4a6e9cce0020b"
server_key_iv: "947fdeb77fbc7edbee3dba8a2c7c9752"
server_key_version: "encryptionkey@0"
I20260811 12:33:13.150381 4922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:13.166851 4922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:13.171387 4922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:13.171689 4922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:13.189949 4922 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:43615
I20260811 12:33:13.189960 5029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:43615 every 8 connection(s)
I20260811 12:33:13.191365 4922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:13.195202 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:13.201159 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 4922
I20260811 12:33:13.201416 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:13.202189 3889 external_mini_cluster.cc:1428] Setting key 55f3a4714ff5d4cf84ee8cc3e6ca2821
I20260811 12:33:13.207243 5030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48: Bootstrap starting.
I20260811 12:33:13.211066 5030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:13.212190 5030 log.cc:824] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:13.215291 5030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48: No bootstrap required, opened a new log
Aug 11 12:33:13 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:13.225380 5030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea50db32da644a0907ea72bf4abad48" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43615 } }
I20260811 12:33:13.225878 5030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:13.225996 5030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea50db32da644a0907ea72bf4abad48, State: Initialized, Role: FOLLOWER
I20260811 12:33:13.226701 5030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea50db32da644a0907ea72bf4abad48" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43615 } }
I20260811 12:33:13.227105 5030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:13.227350 5030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:13.227624 5030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:13.228403 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:13.204676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.3.204.126:43615)
0811 12:33:13.205662 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:13.205680 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:13.206092 (+ 412us) server_negotiation.cc:407] Connection header received
0811 12:33:13.208020 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:13.208046 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:13.208510 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:13.208866 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:13.210703 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:13.212109 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:13.213606 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:13.214287 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:13.214423 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:13.219030 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:13.219041 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:13.219066 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:13.219118 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:13.223263 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:13.223977 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:13.223990 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:13.224000 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:13.224255 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:13.224690 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:13.224695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:13.224698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:13.225487 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:13.225792 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:13.226404 (+ 612us) server_negotiation.cc:299] Negotiation successful
0811 12:33:13.226963 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":398,"threads_started":1}
I20260811 12:33:13.230142 5030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea50db32da644a0907ea72bf4abad48" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43615 } }
I20260811 12:33:13.230643 5030 leader_election.cc:302] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ea50db32da644a0907ea72bf4abad48; no voters:
I20260811 12:33:13.231266 5030 leader_election.cc:288] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:13.231401 5039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:13.232061 5039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea50db32da644a0907ea72bf4abad48, State: Running, Role: LEADER
I20260811 12:33:13.232710 5039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea50db32da644a0907ea72bf4abad48" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43615 } }
I20260811 12:33:13.233655 5030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:13.235896 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ea50db32da644a0907ea72bf4abad48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea50db32da644a0907ea72bf4abad48" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43615 } } }
I20260811 12:33:13.236297 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:13.237048 5043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ea50db32da644a0907ea72bf4abad48. Latest consensus state: current_term: 1 leader_uuid: "5ea50db32da644a0907ea72bf4abad48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea50db32da644a0907ea72bf4abad48" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43615 } } }
I20260811 12:33:13.237424 5043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:13.237720 5058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:13.243412 5058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:13.252750 5058 catalog_manager.cc:1347] Generated new cluster ID: 001bd21371fd4132aa436363c3acafdf
I20260811 12:33:13.252883 5058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:13.266927 5058 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:13.268484 5058 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:13.276510 5058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ea50db32da644a0907ea72bf4abad48: Generated new TSK 0
I20260811 12:33:13.277418 5058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:13.333326 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43615
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:13.675917 5085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:13.676254 5085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:13.676401 5085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:13.683837 5085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:13.683974 5085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:13.684170 5085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:13.694329 5085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:13.695753 5085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:13.697245 5085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:13.711338 5097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:13.711406 5093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:13.711452 5092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:13.712663 5085 server_base.cc:1034] running on GCE node
I20260811 12:33:13.713310 5085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:13.714362 5085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:13.715601 5085 hybrid_clock.cc:648] HybridClock initialized: now 1786451593715538 us; error 49 us; skew 500 ppm
I20260811 12:33:13.715961 5085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:13 dist-test-slave-lgr5 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:13.719300 5085 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:13.720818 5085 webserver.cc:460] Webserver started at http://127.3.204.65:36789/ using document root <none> and password file <none>
I20260811 12:33:13.721511 5085 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:13.721624 5085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:13.721921 5085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:13.725034 5085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "c87d23783f0f4538a6f4eac560735f43"
format_stamp: "Formatted at 2026-08-11 12:33:13 on dist-test-slave-lgr5"
server_key: "4e0771241b9d2ce19359214b509eff5d"
server_key_iv: "06df5cf534442acb1d98d2db03454621"
server_key_version: "encryptionkey@0"
I20260811 12:33:13.725878 5085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "c87d23783f0f4538a6f4eac560735f43"
format_stamp: "Formatted at 2026-08-11 12:33:13 on dist-test-slave-lgr5"
server_key: "4e0771241b9d2ce19359214b509eff5d"
server_key_iv: "06df5cf534442acb1d98d2db03454621"
server_key_version: "encryptionkey@0"
I20260811 12:33:13.731678 5085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:13.735543 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:13.737084 5085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:13.737257 5085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "c87d23783f0f4538a6f4eac560735f43"
format_stamp: "Formatted at 2026-08-11 12:33:13 on dist-test-slave-lgr5"
server_key: "4e0771241b9d2ce19359214b509eff5d"
server_key_iv: "06df5cf534442acb1d98d2db03454621"
server_key_version: "encryptionkey@0"
I20260811 12:33:13.737476 5085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:13.753053 5085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:13.757512 5085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:13.757804 5085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:13.758832 5085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:13.760313 5085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:13.760404 5085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:13.760524 5085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:13.760579 5085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:13.791566 5085 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:44621
I20260811 12:33:13.791606 5319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:44621 every 8 connection(s)
I20260811 12:33:13.792881 5085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:13.796041 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 5085
I20260811 12:33:13.796259 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:13.796849 3889 external_mini_cluster.cc:1428] Setting key 642d5b0e31b706cbb9730b617ab4d577
Aug 11 12:33:13 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:13.813877 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:13.795389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:37459 (local address 127.3.204.126:43615)
0811 12:33:13.795892 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:13.795898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:13.796902 (+ 1004us) server_negotiation.cc:407] Connection header received
0811 12:33:13.799206 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:13.799212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:13.799306 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:13.799435 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:13.800946 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:13.801943 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:13.803594 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:13.804169 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:13.804333 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:13.808000 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:13.808005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:13.808009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:13.808043 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:13.810623 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:13.811398 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:13.811404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:13.811408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:13.811544 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:13.811993 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:13.811997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:13.811999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:13.812326 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:13.812472 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:13.813229 (+ 757us) server_negotiation.cc:299] Negotiation successful
0811 12:33:13.813496 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":251,"threads_started":1}
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
I20260811 12:33:13.814978 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:13.795913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43615 (local address 127.3.204.65:37459)
0811 12:33:13.796719 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:13.796780 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:13.798907 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:13.799608 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:13.799615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:13.800190 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:13.800772 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:13.800791 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:13.802132 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:13.802136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:13.803444 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:13.803457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:13.804543 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:13.804550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:13.804774 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:13.805464 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:13.805489 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:13.807740 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:13.810799 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:13.810807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:13.810823 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:13.811269 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:13.811657 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:13.811661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:13.811663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:13.811870 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:13.812458 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:13.812467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:13.812839 (+ 372us) client_negotiation.cc:769] Sending connection context
0811 12:33:13.813107 (+ 268us) client_negotiation.cc:240] Negotiation successful
0811 12:33:13.813706 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":344,"threads_started":1}
I20260811 12:33:13.816740 5320 heartbeater.cc:347] Connected to a master server at 127.3.204.126:43615
I20260811 12:33:13.817091 5320 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
I20260811 12:33:13.817946 5320 heartbeater.cc:511] Master 127.3.204.126:43615 requested a full tablet report, sending...
I20260811 12:33:13.820253 4975 ts_manager.cc:194] Registered new tserver with Master: c87d23783f0f4538a6f4eac560735f43 (127.3.204.65:44621)
I20260811 12:33:13.821774 4975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:37459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:13.841789 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43615
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:14.175691 5332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:14.175995 5332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:14.176162 5332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:14.183658 5332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:14.183789 5332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:14.183946 5332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:14.194168 5332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:14.195533 5332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:14.196894 5332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:14.210186 5340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:14.210314 5341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:14.210551 5332 server_base.cc:1034] running on GCE node
W20260811 12:33:14.210278 5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:14.211352 5332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:14.212383 5332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:14.213624 5332 hybrid_clock.cc:648] HybridClock initialized: now 1786451594213566 us; error 38 us; skew 500 ppm
I20260811 12:33:14.213977 5332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:14 dist-test-slave-lgr5 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:14.217229 5332 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:14.218787 5332 webserver.cc:460] Webserver started at http://127.3.204.66:36899/ using document root <none> and password file <none>
I20260811 12:33:14.219491 5332 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:14.219609 5332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:14.219929 5332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:14.222939 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "0c2be27eeb444d7d8754135c4373aaa2"
format_stamp: "Formatted at 2026-08-11 12:33:14 on dist-test-slave-lgr5"
server_key: "e0926822d527890cfdd6dcc4df6351e5"
server_key_iv: "352f74091692da68a99ec8d167d5323f"
server_key_version: "encryptionkey@0"
I20260811 12:33:14.223712 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "0c2be27eeb444d7d8754135c4373aaa2"
format_stamp: "Formatted at 2026-08-11 12:33:14 on dist-test-slave-lgr5"
server_key: "e0926822d527890cfdd6dcc4df6351e5"
server_key_iv: "352f74091692da68a99ec8d167d5323f"
server_key_version: "encryptionkey@0"
I20260811 12:33:14.228907 5332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260811 12:33:14.232599 5363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:14.233955 5332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 12:33:14.234112 5332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "0c2be27eeb444d7d8754135c4373aaa2"
format_stamp: "Formatted at 2026-08-11 12:33:14 on dist-test-slave-lgr5"
server_key: "e0926822d527890cfdd6dcc4df6351e5"
server_key_iv: "352f74091692da68a99ec8d167d5323f"
server_key_version: "encryptionkey@0"
I20260811 12:33:14.234328 5332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:14.267633 5332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:14.272146 5332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:14.272436 5332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:14.273451 5332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:14.274861 5332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:14.274950 5332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:14.275091 5332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:14.275147 5332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:14.307825 5332 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:37053
I20260811 12:33:14.307874 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:37053 every 8 connection(s)
I20260811 12:33:14.309192 5332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:14.311844 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 5332
I20260811 12:33:14.312057 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:14.312529 3889 external_mini_cluster.cc:1428] Setting key cab84208ff0da326d7fcf6eef5497bcf
Aug 11 12:33:14 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
I20260811 12:33:14.329414 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.311782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:58983 (local address 127.3.204.126:43615)
0811 12:33:14.312016 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:14.312022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:14.313284 (+ 1262us) server_negotiation.cc:407] Connection header received
0811 12:33:14.315225 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:14.315231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:14.315313 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:14.315399 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:14.317032 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.317749 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.319299 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.319728 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.319841 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.323401 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:14.323408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:14.323411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:14.323446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:14.326166 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:14.327041 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:14.327046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:14.327048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:14.327187 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:14.327636 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:14.327640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:14.327642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:14.327923 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:14.328063 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:14.328830 (+ 767us) server_negotiation.cc:299] Negotiation successful
0811 12:33:14.329076 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260811 12:33:14.330415 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.312327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43615 (local address 127.3.204.66:58983)
0811 12:33:14.313111 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:14.313169 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:14.314945 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:14.315672 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:14.315681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:14.316276 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:14.316863 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.316881 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.317905 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.317909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.319155 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.319168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.319961 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.319969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.320251 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.320921 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:14.320949 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:14.323157 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:14.326328 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:14.326339 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:14.326359 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:14.326904 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:14.327301 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:14.327307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:14.327311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:14.327516 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:14.328053 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:14.328061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:14.328449 (+ 388us) client_negotiation.cc:769] Sending connection context
0811 12:33:14.328707 (+ 258us) client_negotiation.cc:240] Negotiation successful
0811 12:33:14.329284 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":316,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
I20260811 12:33:14.331856 5591 heartbeater.cc:347] Connected to a master server at 127.3.204.126:43615
I20260811 12:33:14.332226 5591 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:14.332947 5591 heartbeater.cc:511] Master 127.3.204.126:43615 requested a full tablet report, sending...
I20260811 12:33:14.334502 4975 ts_manager.cc:194] Registered new tserver with Master: 0c2be27eeb444d7d8754135c4373aaa2 (127.3.204.66:37053)
I20260811 12:33:14.335430 4975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:58983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:14.353103 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43615
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:14.682951 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:14.683247 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:14.683350 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:14.690850 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:14.690972 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:14.691090 5603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:14.701133 5603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:14.702462 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:14.703841 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:14.719863 5610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:14.719966 5613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:14.719988 5611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:14.720811 5603 server_base.cc:1034] running on GCE node
I20260811 12:33:14.721446 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:14.722445 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:14.723649 5603 hybrid_clock.cc:648] HybridClock initialized: now 1786451594723581 us; error 63 us; skew 500 ppm
I20260811 12:33:14.723980 5603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:14 dist-test-slave-lgr5 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:14.727180 5603 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:14.728747 5603 webserver.cc:460] Webserver started at http://127.3.204.67:43079/ using document root <none> and password file <none>
I20260811 12:33:14.729489 5603 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:14.729601 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:14.729928 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:14.733143 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "51c8584cdedb4f81b92d67f1fb6310d1"
format_stamp: "Formatted at 2026-08-11 12:33:14 on dist-test-slave-lgr5"
server_key: "fb8135fef5cda71a5a8486bd39a83161"
server_key_iv: "6be3b5751c9e2344c099bc0538bda22b"
server_key_version: "encryptionkey@0"
I20260811 12:33:14.733984 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "51c8584cdedb4f81b92d67f1fb6310d1"
format_stamp: "Formatted at 2026-08-11 12:33:14 on dist-test-slave-lgr5"
server_key: "fb8135fef5cda71a5a8486bd39a83161"
server_key_iv: "6be3b5751c9e2344c099bc0538bda22b"
server_key_version: "encryptionkey@0"
I20260811 12:33:14.739852 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:14.743505 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:14.745051 5603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:14.745213 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "51c8584cdedb4f81b92d67f1fb6310d1"
format_stamp: "Formatted at 2026-08-11 12:33:14 on dist-test-slave-lgr5"
server_key: "fb8135fef5cda71a5a8486bd39a83161"
server_key_iv: "6be3b5751c9e2344c099bc0538bda22b"
server_key_version: "encryptionkey@0"
I20260811 12:33:14.745422 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:14.769457 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:14.773871 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:14.774151 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:14.775202 5603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:14.776628 5603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:14.776705 5603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:14.776819 5603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:14.776870 5603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:14.808679 5603 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:46261
I20260811 12:33:14.808739 5827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:46261 every 8 connection(s)
I20260811 12:33:14.810009 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:14.812788 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 5603
I20260811 12:33:14.812989 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:14.813405 3889 external_mini_cluster.cc:1428] Setting key d1ab1fd4dfe78d3070aeac9713821b4b
Aug 11 12:33:14 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:14.824991 5320 heartbeater.cc:503] Master 127.3.204.126:43615 was elected leader, sending a full tablet report...
I20260811 12:33:14.831830 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.812740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:41965 (local address 127.3.204.126:43615)
0811 12:33:14.812960 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:14.812966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:14.814227 (+ 1261us) server_negotiation.cc:407] Connection header received
0811 12:33:14.816183 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:14.816188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:14.816306 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:14.816394 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:14.818100 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.818885 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.820620 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.821037 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.821153 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.824868 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:14.824873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:14.824877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:14.824913 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:14.827973 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:14.829133 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:14.829138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:14.829141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:14.829318 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:14.829902 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:14.829908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:14.829910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:14.830211 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:14.830329 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:14.831211 (+ 882us) server_negotiation.cc:299] Negotiation successful
0811 12:33:14.831472 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 12:33:14.832948 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.813201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43615 (local address 127.3.204.67:41965)
0811 12:33:14.814044 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:14.814108 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:14.815869 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:14.816659 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:14.816668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:14.817206 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:14.817821 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.817848 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.819065 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.819069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.820466 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.820481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.821276 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.821284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.821535 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.822222 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:14.822248 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:14.824619 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:14.828182 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:14.828190 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:14.828206 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:14.828890 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:14.829530 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:14.829534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:14.829536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:14.829758 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:14.830354 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:14.830362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:14.830771 (+ 409us) client_negotiation.cc:769] Sending connection context
0811 12:33:14.831061 (+ 290us) client_negotiation.cc:240] Negotiation successful
0811 12:33:14.831723 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":321,"threads_started":1}
I20260811 12:33:14.834564 5829 heartbeater.cc:347] Connected to a master server at 127.3.204.126:43615
I20260811 12:33:14.834918 5829 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:14.835750 5829 heartbeater.cc:511] Master 127.3.204.126:43615 requested a full tablet report, sending...
I20260811 12:33:14.837474 4975 ts_manager.cc:194] Registered new tserver with Master: 51c8584cdedb4f81b92d67f1fb6310d1 (127.3.204.67:46261)
I20260811 12:33:14.838383 4975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:41965
I20260811 12:33:14.842046 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:33:14.865511 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.849811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.3.204.126:43615)
0811 12:33:14.850014 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:14.850020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:14.850538 (+ 518us) server_negotiation.cc:407] Connection header received
0811 12:33:14.850709 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:14.850714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:14.850788 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:14.850890 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:14.853306 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.854234 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.855720 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.856199 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.856305 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.858877 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:14.858882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:14.858886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:14.858924 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:14.862234 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:14.863115 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:14.863120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:14.863123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:14.863302 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:14.863702 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:14.863705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:14.863707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:14.864121 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:14.864402 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:14.864883 (+ 481us) server_negotiation.cc:299] Negotiation successful
0811 12:33:14.865123 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:33:14.875344 4975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:14.885095 4975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:14.921072 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.904355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.3.204.66:37053)
0811 12:33:14.911556 (+ 7201us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:14.911564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:14.911599 (+ 35us) server_negotiation.cc:407] Connection header received
0811 12:33:14.911708 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:14.911717 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:14.912047 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:14.912280 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:14.914167 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.915429 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.917958 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.919586 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.919736 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.919965 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:14.920573 (+ 608us) server_negotiation.cc:299] Negotiation successful
0811 12:33:14.920761 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6971,"thread_start_us":8138,"threads_started":1}
I20260811 12:33:14.921701 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.903745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:37053 (local address 127.0.0.1:60430)
0811 12:33:14.909888 (+ 6143us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:14.909935 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:14.910183 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:14.913630 (+ 3447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:14.913635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:14.913660 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:14.914004 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.914013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.915549 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.915553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.917791 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.917805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.919895 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.919904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.920220 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.920324 (+ 104us) client_negotiation.cc:769] Sending connection context
0811 12:33:14.920484 (+ 160us) client_negotiation.cc:240] Negotiation successful
0811 12:33:14.920692 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5839,"mutex_wait_us":2,"thread_start_us":5617,"threads_started":1}
I20260811 12:33:14.922355 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.909563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:46261 (local address 127.0.0.1:43476)
0811 12:33:14.913358 (+ 3795us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:14.913376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:14.913499 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:14.914447 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:14.914452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:14.914478 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:14.914859 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.914872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.917087 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.917094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.919142 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.919156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.921458 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.921467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.921650 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.921693 (+ 43us) client_negotiation.cc:769] Sending connection context
0811 12:33:14.921791 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 12:33:14.921933 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3642,"mutex_wait_us":1378,"thread_start_us":317,"threads_started":1}
I20260811 12:33:14.922714 5852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.910263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.3.204.67:46261)
0811 12:33:14.913848 (+ 3585us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:14.913856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:14.913884 (+ 28us) server_negotiation.cc:407] Connection header received
0811 12:33:14.913987 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:14.913994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:14.914221 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:14.914437 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:14.915061 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.916722 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.919296 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.920837 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.920990 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.921224 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:14.922046 (+ 822us) server_negotiation.cc:299] Negotiation successful
0811 12:33:14.922218 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3427,"thread_start_us":3242,"threads_started":1}
I20260811 12:33:14.922873 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.903918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44621 (local address 127.0.0.1:37622)
0811 12:33:14.911567 (+ 7649us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:14.911584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:14.911789 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:14.912733 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:14.912745 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:14.912773 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:14.913691 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.913705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.915619 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.915625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.918097 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:14.918115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.920266 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.920275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:14.921909 (+ 1634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.921944 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:33:14.922044 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 12:33:14.922180 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7523,"mutex_wait_us":1,"thread_start_us":9260,"threads_started":1}
I20260811 12:33:14.922957 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:14.903965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.3.204.65:44621)
0811 12:33:14.911039 (+ 7074us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:14.911051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:14.911685 (+ 634us) server_negotiation.cc:407] Connection header received
0811 12:33:14.911974 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:14.911983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:14.912255 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:14.912419 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:14.913905 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.915423 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.918277 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:14.919944 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:14.920115 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:14.920351 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:14.922461 (+ 2110us) server_negotiation.cc:299] Negotiation successful
0811 12:33:14.922641 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6806,"thread_start_us":6576,"threads_started":1}
I20260811 12:33:14.926573 5706 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1693aa3d7d40ce8d4572d6e06bc3b8 (DEFAULT_TABLE table=test-table [id=fba7ea3bcc77448f99408baebb50cb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:14.926573 5201 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1693aa3d7d40ce8d4572d6e06bc3b8 (DEFAULT_TABLE table=test-table [id=fba7ea3bcc77448f99408baebb50cb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:14.926574 5458 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1693aa3d7d40ce8d4572d6e06bc3b8 (DEFAULT_TABLE table=test-table [id=fba7ea3bcc77448f99408baebb50cb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:14.928508 5201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1693aa3d7d40ce8d4572d6e06bc3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:14.928506 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1693aa3d7d40ce8d4572d6e06bc3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:14.930747 5706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1693aa3d7d40ce8d4572d6e06bc3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:14.943575 5857 tablet_bootstrap.cc:492] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2: Bootstrap starting.
I20260811 12:33:14.946270 5859 tablet_bootstrap.cc:492] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43: Bootstrap starting.
I20260811 12:33:14.948037 5857 tablet_bootstrap.cc:654] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:14.948702 5860 tablet_bootstrap.cc:492] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1: Bootstrap starting.
I20260811 12:33:14.950786 5859 tablet_bootstrap.cc:654] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:14.950951 5857 log.cc:824] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:14.952109 5859 log.cc:824] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:14.953158 5860 tablet_bootstrap.cc:654] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:14.954594 5860 log.cc:824] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:14.954718 5857 tablet_bootstrap.cc:492] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2: No bootstrap required, opened a new log
I20260811 12:33:14.954972 5857 ts_tablet_manager.cc:1397] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260811 12:33:14.956271 5859 tablet_bootstrap.cc:492] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43: No bootstrap required, opened a new log
I20260811 12:33:14.956681 5859 ts_tablet_manager.cc:1397] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260811 12:33:14.958606 5860 tablet_bootstrap.cc:492] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1: No bootstrap required, opened a new log
I20260811 12:33:14.958930 5860 ts_tablet_manager.cc:1397] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260811 12:33:14.963024 5857 raft_consensus.cc:348] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:14.963423 5857 raft_consensus.cc:374] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:14.963536 5857 raft_consensus.cc:729] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2be27eeb444d7d8754135c4373aaa2, State: Initialized, Role: FOLLOWER
I20260811 12:33:14.964190 5857 consensus_queue.cc:260] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:14.965806 5591 heartbeater.cc:503] Master 127.3.204.126:43615 was elected leader, sending a full tablet report...
I20260811 12:33:14.966388 5857 ts_tablet_manager.cc:1428] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260811 12:33:14.970837 5859 raft_consensus.cc:348] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:14.971395 5859 raft_consensus.cc:374] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:14.971552 5859 raft_consensus.cc:729] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87d23783f0f4538a6f4eac560735f43, State: Initialized, Role: FOLLOWER
I20260811 12:33:14.971135 5860 raft_consensus.cc:348] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:14.971805 5860 raft_consensus.cc:374] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:14.971895 5860 raft_consensus.cc:729] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c8584cdedb4f81b92d67f1fb6310d1, State: Initialized, Role: FOLLOWER
I20260811 12:33:14.972543 5859 consensus_queue.cc:260] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:14.972770 5860 consensus_queue.cc:260] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:14.974902 5829 heartbeater.cc:503] Master 127.3.204.126:43615 was elected leader, sending a full tablet report...
I20260811 12:33:14.975054 5859 ts_tablet_manager.cc:1428] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43: Time spent starting tablet: real 0.018s user 0.014s sys 0.001s
I20260811 12:33:14.976353 5860 ts_tablet_manager.cc:1428] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1: Time spent starting tablet: real 0.017s user 0.009s sys 0.010s
W20260811 12:33:15.045361 5321 tablet.cc:2369] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:33:15.061331 5593 tablet.cc:2369] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:33:15.061851 5830 tablet.cc:2369] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:33:15.108446 5872 raft_consensus.cc:482] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:15.108973 5872 raft_consensus.cc:504] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:15.111652 5872 leader_election.cc:288] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87d23783f0f4538a6f4eac560735f43 (127.3.204.65:44621), 0c2be27eeb444d7d8754135c4373aaa2 (127.3.204.66:37053)
I20260811 12:33:15.118127 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.112068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:39057 (local address 127.3.204.66:37053)
0811 12:33:15.112351 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:15.112359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:15.112501 (+ 142us) server_negotiation.cc:407] Connection header received
0811 12:33:15.112817 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:15.112823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:15.112906 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:15.112992 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:15.113561 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.114421 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.116126 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.117085 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.117249 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.117339 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:15.117737 (+ 398us) server_negotiation.cc:299] Negotiation successful
0811 12:33:15.117871 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260811 12:33:15.118127 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.112001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:37053 (local address 127.3.204.67:39057)
0811 12:33:15.112389 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:15.112411 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:15.112571 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:15.113190 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:15.113195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:15.113226 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:15.113420 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:15.113431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.114585 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.114589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:15.115937 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:15.115950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.117237 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.117243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:15.117572 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.117600 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 12:33:15.117685 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 12:33:15.117850 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":40}
I20260811 12:33:15.119465 5491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1693aa3d7d40ce8d4572d6e06bc3b8" candidate_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2be27eeb444d7d8754135c4373aaa2" is_pre_election: true
I20260811 12:33:15.120434 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.112442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:43331 (local address 127.3.204.65:44621)
0811 12:33:15.112743 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:15.112751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:15.113912 (+ 1161us) server_negotiation.cc:407] Connection header received
0811 12:33:15.114086 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:15.114093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:15.114160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:15.114249 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:15.115281 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.116223 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.117957 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.119331 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.119506 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.119592 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:15.119878 (+ 286us) server_negotiation.cc:299] Negotiation successful
0811 12:33:15.120015 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 12:33:15.120437 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.112135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44621 (local address 127.3.204.67:43331)
0811 12:33:15.113791 (+ 1656us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:15.113812 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:15.113949 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:15.114282 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:15.114286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:15.114311 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:15.115082 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:15.115095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.116378 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.116383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:15.117781 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:15.117796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.119542 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.119550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:15.119705 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.119731 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 12:33:15.119836 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 12:33:15.119979 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"mutex_wait_us":1,"thread_start_us":627,"threads_started":1}
I20260811 12:33:15.120529 5491 raft_consensus.cc:2454] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51c8584cdedb4f81b92d67f1fb6310d1 in term 0.
I20260811 12:33:15.121503 5628 leader_election.cc:302] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2be27eeb444d7d8754135c4373aaa2, 51c8584cdedb4f81b92d67f1fb6310d1; no voters:
I20260811 12:33:15.121778 5227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1693aa3d7d40ce8d4572d6e06bc3b8" candidate_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87d23783f0f4538a6f4eac560735f43" is_pre_election: true
I20260811 12:33:15.122032 5872 raft_consensus.cc:2781] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:15.122352 5872 raft_consensus.cc:482] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:15.122387 5227 raft_consensus.cc:2454] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51c8584cdedb4f81b92d67f1fb6310d1 in term 0.
I20260811 12:33:15.122560 5872 raft_consensus.cc:3037] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:15.126753 5872 raft_consensus.cc:504] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:15.127753 5872 leader_election.cc:288] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [CANDIDATE]: Term 1 election: Requested vote from peers c87d23783f0f4538a6f4eac560735f43 (127.3.204.65:44621), 0c2be27eeb444d7d8754135c4373aaa2 (127.3.204.66:37053)
I20260811 12:33:15.128307 5227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1693aa3d7d40ce8d4572d6e06bc3b8" candidate_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87d23783f0f4538a6f4eac560735f43"
I20260811 12:33:15.128343 5491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1693aa3d7d40ce8d4572d6e06bc3b8" candidate_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2be27eeb444d7d8754135c4373aaa2"
I20260811 12:33:15.128553 5491 raft_consensus.cc:3037] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:15.128616 5227 raft_consensus.cc:3037] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:15.130932 5491 raft_consensus.cc:2454] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 0c2be27eeb444d7d8754135c4373aaa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51c8584cdedb4f81b92d67f1fb6310d1 in term 1.
I20260811 12:33:15.131006 5227 raft_consensus.cc:2454] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P c87d23783f0f4538a6f4eac560735f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51c8584cdedb4f81b92d67f1fb6310d1 in term 1.
I20260811 12:33:15.131525 5628 leader_election.cc:302] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2be27eeb444d7d8754135c4373aaa2, 51c8584cdedb4f81b92d67f1fb6310d1; no voters:
I20260811 12:33:15.131860 5872 raft_consensus.cc:2781] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:15.132537 5872 raft_consensus.cc:686] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [term 1 LEADER]: Becoming Leader. State: Replica: 51c8584cdedb4f81b92d67f1fb6310d1, State: Running, Role: LEADER
I20260811 12:33:15.133088 5872 consensus_queue.cc:237] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } }
I20260811 12:33:15.139215 4973 catalog_manager.cc:5495] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51c8584cdedb4f81b92d67f1fb6310d1 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87d23783f0f4538a6f4eac560735f43" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c2be27eeb444d7d8754135c4373aaa2" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51c8584cdedb4f81b92d67f1fb6310d1" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 46261 } health_report { overall_health: HEALTHY } } }
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:15.171613 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.152413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.3.204.65:44621)
0811 12:33:15.152623 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:15.152629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:15.153575 (+ 946us) server_negotiation.cc:407] Connection header received
0811 12:33:15.153768 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:15.153772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:15.153853 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:15.154003 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:15.156166 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.157016 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.158621 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.159082 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.159204 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.163887 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:15.163906 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:15.163925 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:15.163970 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:15.167723 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.168718 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.168727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.168729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.168904 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.169693 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.169698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.169701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.170335 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:15.170740 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:15.170964 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 12:33:15.171171 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:15.230906 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.214779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.3.204.65:44621)
0811 12:33:15.215040 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:15.215048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:15.215505 (+ 457us) server_negotiation.cc:407] Connection header received
0811 12:33:15.215727 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:15.215732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:15.215805 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:15.215912 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:15.217555 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.218345 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.220018 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.220493 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.220600 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.225107 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:15.225113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:15.225117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:15.225161 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:15.227943 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.228757 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.228761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.228764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.228957 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.229407 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.229410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.229413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.229755 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:15.230060 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:15.230338 (+ 278us) server_negotiation.cc:299] Negotiation successful
0811 12:33:15.230564 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:15.284231 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.268129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.3.204.65:44621)
0811 12:33:15.268373 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:15.268380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:15.268632 (+ 252us) server_negotiation.cc:407] Connection header received
0811 12:33:15.268783 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:15.268786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:15.268858 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:15.268950 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:15.271128 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.271848 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.273114 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.273554 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.273646 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.277984 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:15.277991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:15.277994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:15.278033 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:15.280973 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.281805 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.281810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.281813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.281971 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.282452 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.282457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.282459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.282827 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:15.283050 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:15.283532 (+ 482us) server_negotiation.cc:299] Negotiation successful
0811 12:33:15.283746 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260811 12:33:15.286886 5161 tablet_service.cc:2908] Rejecting scan request for tablet 9d1693aa3d7d40ce8d4572d6e06bc3b8: Uninitialized: clean time has not yet been initialized
I20260811 12:33:15.290751 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 5085
I20260811 12:33:15.305260 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.291713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.3.204.65:44621)
0811 12:33:15.291936 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:15.291941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:15.292553 (+ 612us) server_negotiation.cc:407] Connection header received
0811 12:33:15.292719 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:15.292724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:15.292811 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:15.292899 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:15.294429 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.295178 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.296606 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.297058 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.297203 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.299428 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:15.299433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:15.299437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:15.299479 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:15.302177 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.303048 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.303053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.303055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.303242 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.303681 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.303686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.303690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.304058 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:15.304386 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:15.304692 (+ 306us) server_negotiation.cc:299] Negotiation successful
0811 12:33:15.304939 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 12:33:15.306136 5312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:15.405932 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 5085
I20260811 12:33:15.429702 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 5332
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:33:15.445870 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.430705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.3.204.66:37053)
0811 12:33:15.430940 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:15.430946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:15.430966 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:33:15.431027 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:15.431031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:15.431094 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:15.431192 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:15.432674 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.433470 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.434838 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.435300 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.435427 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.439811 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:15.439829 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:15.439848 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:15.439905 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:15.442751 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.443545 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.443554 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.443557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.443707 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.444351 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.444355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.444358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.444848 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:15.445128 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:15.445392 (+ 264us) server_negotiation.cc:299] Negotiation successful
0811 12:33:15.445606 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260811 12:33:15.446650 5581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:15.535203 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 5332
I20260811 12:33:15.554368 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 5603
W20260811 12:33:15.554191 5628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.66:37053 (error 108)
I20260811 12:33:15.556484 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.555887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:37053 (local address 127.3.204.67:38129)
0811 12:33:15.556069 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:15.556282 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.66:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260811 12:33:15.556793 5628 consensus_peers.cc:487] T 9d1693aa3d7d40ce8d4572d6e06bc3b8 P 51c8584cdedb4f81b92d67f1fb6310d1 -> Peer 0c2be27eeb444d7d8754135c4373aaa2 (127.3.204.66:37053): Couldn't send request to peer 0c2be27eeb444d7d8754135c4373aaa2. Status: Network error: Client connection negotiation failed: client connection to 127.3.204.66:37053: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:33:15.570562 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:15.555452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.3.204.67:46261)
0811 12:33:15.555909 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:15.555916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:15.555944 (+ 28us) server_negotiation.cc:407] Connection header received
0811 12:33:15.556016 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:15.556023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:15.556128 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:15.556245 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:15.557589 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.558319 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.559589 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:15.560025 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:15.560268 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:15.564446 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:15.564461 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:15.564479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:15.564525 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:15.567426 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.568218 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.568228 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.568230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.568410 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:15.568844 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:15.568849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:15.568852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:15.569271 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:15.569579 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:15.569904 (+ 325us) server_negotiation.cc:299] Negotiation successful
0811 12:33:15.570149 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":561,"threads_started":1}
I20260811 12:33:15.571285 5820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:15.660368 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 5603
I20260811 12:33:15.682890 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 4922
I20260811 12:33:15.684139 5024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:15.793696 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 4922
2026-08-11T12:33:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5710 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[5916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[5916](info): set up 2 sockets
Aug 11 12:33:15 dist-test-slave-lgr5 krb5kdc[5916](info): commencing operation
krb5kdc: starting...
W20260811 12:33:17.053562 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:17 dist-test-slave-lgr5 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:17.255522 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33047
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:33047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:17.585394 5938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:17.585693 5938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:17.585795 5938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:17.593048 5938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:17.593176 5938 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 12:33:17.593216 5938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:17.593292 5938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:17.593369 5938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:17.593449 5938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:17.603487 5938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:33047
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:17.604868 5938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:17.606181 5938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:17.618045 5945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:17.618170 5953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:17.618193 5946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:17.618762 5938 server_base.cc:1034] running on GCE node
I20260811 12:33:17.619503 5938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:17.620949 5938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:17.622169 5938 hybrid_clock.cc:648] HybridClock initialized: now 1786451597622110 us; error 36 us; skew 500 ppm
I20260811 12:33:17.622495 5938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:17 dist-test-slave-lgr5 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:17.625792 5938 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:17.627347 5938 webserver.cc:460] Webserver started at http://127.3.204.126:46215/ using document root <none> and password file <none>
I20260811 12:33:17.628109 5938 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:17.628226 5938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:17.628552 5938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:17.631614 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8"
format_stamp: "Formatted at 2026-08-11 12:33:17 on dist-test-slave-lgr5"
server_key: "adb77dabbe28ee29efa9d5b820c17885"
server_key_iv: "f161337e9f1bffb9ef9c4ef40041d116"
server_key_version: "encryptionkey@0"
I20260811 12:33:17.632481 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8"
format_stamp: "Formatted at 2026-08-11 12:33:17 on dist-test-slave-lgr5"
server_key: "adb77dabbe28ee29efa9d5b820c17885"
server_key_iv: "f161337e9f1bffb9ef9c4ef40041d116"
server_key_version: "encryptionkey@0"
I20260811 12:33:17.637909 5938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 12:33:17.641844 5967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:17.643380 5938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260811 12:33:17.643585 5938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8"
format_stamp: "Formatted at 2026-08-11 12:33:17 on dist-test-slave-lgr5"
server_key: "adb77dabbe28ee29efa9d5b820c17885"
server_key_iv: "f161337e9f1bffb9ef9c4ef40041d116"
server_key_version: "encryptionkey@0"
I20260811 12:33:17.643819 5938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:17.664428 5938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:17.668943 5938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:17.669206 5938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:17.690375 5938 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:33047
I20260811 12:33:17.690383 6069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:33047 every 8 connection(s)
I20260811 12:33:17.691758 5938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:17.695601 6070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:17.695636 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 5938
I20260811 12:33:17.695838 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:17.696571 3889 external_mini_cluster.cc:1428] Setting key 879d57819402c403c583ff920aeb52af
I20260811 12:33:17.709363 6070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8: Bootstrap starting.
Aug 11 12:33:17 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:17.713413 6070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:17.714838 6070 log.cc:824] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:17.718912 6070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8: No bootstrap required, opened a new log
I20260811 12:33:17.721349 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:17.698485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.3.204.126:33047)
0811 12:33:17.699281 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:17.699293 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:17.699336 (+ 43us) server_negotiation.cc:407] Connection header received
0811 12:33:17.701513 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:17.701543 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:17.702038 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:17.702416 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:17.704239 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:17.705605 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:17.707357 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:17.708045 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:17.708237 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:17.713041 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:17.713050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:17.713067 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:17.713116 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:17.717174 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:17.718064 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:17.718070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:17.718077 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:17.718258 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:17.718660 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:17.718663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:17.718665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:17.719216 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:17.719475 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:17.719881 (+ 406us) server_negotiation.cc:299] Negotiation successful
0811 12:33:17.720277 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":373,"threads_started":1}
I20260811 12:33:17.727592 6070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33047 } }
I20260811 12:33:17.727968 6070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:17.728070 6070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188cc6c25ddf4a2cacf47ffffb3f0cc8, State: Initialized, Role: FOLLOWER
I20260811 12:33:17.728693 6070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33047 } }
I20260811 12:33:17.728925 6070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:17.729054 6070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:17.729218 6070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:17.731607 6070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33047 } }
I20260811 12:33:17.732153 6070 leader_election.cc:302] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 188cc6c25ddf4a2cacf47ffffb3f0cc8; no voters:
I20260811 12:33:17.732744 6070 leader_election.cc:288] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:17.732864 6077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:17.733582 6077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [term 1 LEADER]: Becoming Leader. State: Replica: 188cc6c25ddf4a2cacf47ffffb3f0cc8, State: Running, Role: LEADER
I20260811 12:33:17.734185 6077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33047 } }
I20260811 12:33:17.734781 6070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:17.737222 6079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33047 } } }
I20260811 12:33:17.737629 6079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:17.738180 6081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 188cc6c25ddf4a2cacf47ffffb3f0cc8. Latest consensus state: current_term: 1 leader_uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "188cc6c25ddf4a2cacf47ffffb3f0cc8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33047 } } }
I20260811 12:33:17.739328 6081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:17.739626 6091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:17.745604 6091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:17.754335 6091 catalog_manager.cc:1347] Generated new cluster ID: 533cff1dc6e6429abcf4970905cc8d91
I20260811 12:33:17.754469 6091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:17.780400 6091 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:17.782367 6091 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:17.796571 6091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8: Generated new TSK 0
I20260811 12:33:17.797758 6091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:17.854090 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33047
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:18.193892 6115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:18.194196 6115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:18.194324 6115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:18.201737 6115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:18.201864 6115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:18.201972 6115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:33:18.202085 6115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:18.212177 6115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:33047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:18.213694 6115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:18.215071 6115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:18.228879 6127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:18.228847 6130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:18.228912 6128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:18.229743 6115 server_base.cc:1034] running on GCE node
I20260811 12:33:18.230360 6115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:18.231345 6115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:18.232558 6115 hybrid_clock.cc:648] HybridClock initialized: now 1786451598232483 us; error 56 us; skew 500 ppm
I20260811 12:33:18.232863 6115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:18 dist-test-slave-lgr5 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:18.239876 6115 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:18.241494 6115 webserver.cc:460] Webserver started at http://127.3.204.65:41001/ using document root <none> and password file <none>
I20260811 12:33:18.242254 6115 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:18.242367 6115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:18.242657 6115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:18.245750 6115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "b3058bac42ec4f7286289fde5e1d43ce"
format_stamp: "Formatted at 2026-08-11 12:33:18 on dist-test-slave-lgr5"
server_key: "6667983221050d662fcf12daa2263e30"
server_key_iv: "f4fcc19a6320844166051ead585aef48"
server_key_version: "encryptionkey@0"
I20260811 12:33:18.246524 6115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "b3058bac42ec4f7286289fde5e1d43ce"
format_stamp: "Formatted at 2026-08-11 12:33:18 on dist-test-slave-lgr5"
server_key: "6667983221050d662fcf12daa2263e30"
server_key_iv: "f4fcc19a6320844166051ead585aef48"
server_key_version: "encryptionkey@0"
I20260811 12:33:18.251907 6115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:33:18.255550 6140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:18.256995 6115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:18.257153 6115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "b3058bac42ec4f7286289fde5e1d43ce"
format_stamp: "Formatted at 2026-08-11 12:33:18 on dist-test-slave-lgr5"
server_key: "6667983221050d662fcf12daa2263e30"
server_key_iv: "f4fcc19a6320844166051ead585aef48"
server_key_version: "encryptionkey@0"
I20260811 12:33:18.257400 6115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:18.287055 6115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:18.291543 6115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:18.291838 6115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:18.294893 6115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:18.295019 6115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:18.295171 6115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:18.295248 6115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:33:18 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:18.320361 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:18.295849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:56385 (local address 127.3.204.126:33047)
0811 12:33:18.296431 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:18.296436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:18.297459 (+ 1023us) server_negotiation.cc:407] Connection header received
0811 12:33:18.299590 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:18.299596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:18.299695 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:18.299794 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:18.302164 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.303326 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.305781 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.306384 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.306523 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:18.312391 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:18.312397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:18.312401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:18.312442 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:18.315981 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:18.317205 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:18.317209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:18.317211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:18.317334 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:18.317939 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:18.317953 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:18.317957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:18.318331 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:18.318481 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:18.319620 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0811 12:33:18.319921 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":260,"threads_started":1}
I20260811 12:33:18.321270 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:18.296454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33047 (local address 127.3.204.65:56385)
0811 12:33:18.297274 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:18.297326 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:18.299246 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:18.300136 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:18.300151 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:18.300997 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:18.301919 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:18.301954 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.303522 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.303528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:18.305628 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:18.305642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.306691 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.306701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:18.307207 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:18.308341 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:18.308376 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:18.312039 (+ 3663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:18.316207 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:18.316217 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:18.316244 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:18.317023 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:18.317447 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:18.317456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:18.317459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:18.317766 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:18.318562 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:18.318568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:18.319116 (+ 548us) client_negotiation.cc:769] Sending connection context
0811 12:33:18.319464 (+ 348us) client_negotiation.cc:240] Negotiation successful
0811 12:33:18.319965 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":342,"threads_started":1}
I20260811 12:33:18.343377 6115 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:38679
I20260811 12:33:18.343398 6362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:38679 every 8 connection(s)
I20260811 12:33:18.345098 6115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:18.346109 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 6115
I20260811 12:33:18.346351 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:18.347139 3889 external_mini_cluster.cc:1428] Setting key 4c4db2180b2f274c05e538f0880c141a
I20260811 12:33:18.347662 6363 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33047
I20260811 12:33:18.347935 6363 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:18.348938 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:18.331066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:35989 (local address 127.3.204.126:33047)
0811 12:33:18.331256 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:18.331263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:18.332849 (+ 1586us) server_negotiation.cc:407] Connection header received
0811 12:33:18.333243 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:18.333249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:18.333348 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:18.333472 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:18.334929 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.335899 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.338297 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.338959 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.339092 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:18.341738 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:18.341746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:18.341750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:18.341818 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:18.345214 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:18.346198 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:18.346205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:18.346209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:18.346398 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:18.346855 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:18.346861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:18.346864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:18.347353 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:18.347513 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:18.348240 (+ 727us) server_negotiation.cc:299] Negotiation successful
0811 12:33:18.348515 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 12:33:18.348953 6363 heartbeater.cc:511] Master 127.3.204.126:33047 requested a full tablet report, sending...
I20260811 12:33:18.350137 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:18.331121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33047 (local address 127.3.204.65:35989)
0811 12:33:18.332694 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:18.332716 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:18.333015 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:18.333539 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:18.333545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:18.334302 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:18.334734 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:18.334751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.336154 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.336160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:18.338123 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:18.338146 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.339222 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.339229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:18.339429 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:18.340255 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:18.340284 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:18.341492 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:18.345383 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:18.345389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:18.345393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:18.346034 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:18.346550 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:18.346556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:18.346560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:18.346726 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:18.347637 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:18.347646 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:18.347877 (+ 231us) client_negotiation.cc:769] Sending connection context
0811 12:33:18.349208 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0811 12:33:18.349616 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":1189,"threads_started":1}
I20260811 12:33:18.351347 6000 ts_manager.cc:194] Registered new tserver with Master: b3058bac42ec4f7286289fde5e1d43ce (127.3.204.65:38679)
I20260811 12:33:18.354621 6000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:56385
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:18.396374 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33047
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:18.730418 6367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:18.730727 6367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:18.730872 6367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:18.738384 6367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:18.738513 6367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:18.738631 6367 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:33:18.738740 6367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:18.749234 6367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:33047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:18.750564 6367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:18.751919 6367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:18.765163 6377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:18.765132 6376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:18.765193 6380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:18.766448 6367 server_base.cc:1034] running on GCE node
I20260811 12:33:18.767069 6367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:18.768041 6367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:18.769299 6367 hybrid_clock.cc:648] HybridClock initialized: now 1786451598769229 us; error 37 us; skew 500 ppm
I20260811 12:33:18.769594 6367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:18 dist-test-slave-lgr5 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:18.772734 6367 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:18.774196 6367 webserver.cc:460] Webserver started at http://127.3.204.66:40867/ using document root <none> and password file <none>
I20260811 12:33:18.774978 6367 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:18.775089 6367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:18.775440 6367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:18.778503 6367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "5afe16cda3a84d59b8c0c2d6b553ac13"
format_stamp: "Formatted at 2026-08-11 12:33:18 on dist-test-slave-lgr5"
server_key: "050f873c4d12f2067e5a4a815da14e83"
server_key_iv: "3424ace7c9d043df8522a627419389c4"
server_key_version: "encryptionkey@0"
I20260811 12:33:18.779304 6367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "5afe16cda3a84d59b8c0c2d6b553ac13"
format_stamp: "Formatted at 2026-08-11 12:33:18 on dist-test-slave-lgr5"
server_key: "050f873c4d12f2067e5a4a815da14e83"
server_key_iv: "3424ace7c9d043df8522a627419389c4"
server_key_version: "encryptionkey@0"
I20260811 12:33:18.784643 6367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:33:18.788288 6394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:18.789673 6367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:33:18.789824 6367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "5afe16cda3a84d59b8c0c2d6b553ac13"
format_stamp: "Formatted at 2026-08-11 12:33:18 on dist-test-slave-lgr5"
server_key: "050f873c4d12f2067e5a4a815da14e83"
server_key_iv: "3424ace7c9d043df8522a627419389c4"
server_key_version: "encryptionkey@0"
I20260811 12:33:18.790045 6367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:18.817572 6367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:18.821962 6367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:18.822271 6367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:18.825201 6367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:18.825295 6367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:18.825446 6367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:18.825520 6367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:33:18 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:18.850613 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:18.825997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:60103 (local address 127.3.204.126:33047)
0811 12:33:18.826179 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:18.826185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:18.827720 (+ 1535us) server_negotiation.cc:407] Connection header received
0811 12:33:18.830621 (+ 2901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:18.830628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:18.830728 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:18.830847 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:18.833537 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.834483 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.837496 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.838119 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.838256 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:18.843332 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:18.843338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:18.843341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:18.843380 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:18.846550 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:18.847599 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:18.847605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:18.847609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:18.847811 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:18.848435 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:18.848441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:18.848445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:18.848882 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:18.849080 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:18.850016 (+ 936us) server_negotiation.cc:299] Negotiation successful
0811 12:33:18.850251 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 12:33:18.852183 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:18.826579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33047 (local address 127.3.204.66:60103)
0811 12:33:18.827500 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:18.827566 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:18.830251 (+ 2685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:18.831183 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:18.831196 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:18.832175 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:18.833276 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:18.833309 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.834656 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.834667 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:18.837292 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:18.837313 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.838411 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.838420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:18.838668 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:18.839494 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:18.839536 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:18.843008 (+ 3472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:18.846700 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:18.846710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:18.846737 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:18.847436 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:18.847949 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:18.847958 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:18.847961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:18.848298 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:18.849026 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:18.849033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:18.849524 (+ 491us) client_negotiation.cc:769] Sending connection context
0811 12:33:18.849910 (+ 386us) client_negotiation.cc:240] Negotiation successful
0811 12:33:18.850534 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":365,"threads_started":1}
I20260811 12:33:18.871040 6367 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:33493
I20260811 12:33:18.871061 6614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:33493 every 8 connection(s)
I20260811 12:33:18.872216 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:18.858514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:42637 (local address 127.3.204.126:33047)
0811 12:33:18.858680 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:18.858685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:18.858848 (+ 163us) server_negotiation.cc:407] Connection header received
0811 12:33:18.859047 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:18.859053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:18.859143 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:18.859242 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:18.860140 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.861274 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.862651 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.863347 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.863476 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:18.865794 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:18.865800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:18.865804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:18.865856 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:18.869422 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:18.870113 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:18.870118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:18.870122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:18.870307 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:18.870678 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:18.870683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:18.870687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:18.871037 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:18.871174 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:18.871617 (+ 443us) server_negotiation.cc:299] Negotiation successful
0811 12:33:18.871842 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:33:18.872334 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:18.858101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33047 (local address 127.3.204.66:42637)
0811 12:33:18.858709 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:18.858733 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:18.858896 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:18.859287 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:18.859290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:18.859709 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:18.859991 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:18.860000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.861412 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.861418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:18.862497 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:18.862511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:18.863544 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:18.863550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:18.863678 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:18.864449 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:18.864472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:18.865602 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:18.869577 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:18.869581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:18.869584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:18.869990 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:18.870424 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:18.870429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:18.870433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:18.870573 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:18.871155 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:18.871160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:18.871370 (+ 210us) client_negotiation.cc:769] Sending connection context
0811 12:33:18.871503 (+ 133us) client_negotiation.cc:240] Negotiation successful
0811 12:33:18.871863 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":576,"threads_started":1}
I20260811 12:33:18.872664 6367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:18.876659 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 6367
I20260811 12:33:18.876852 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:18.877286 3889 external_mini_cluster.cc:1428] Setting key 2f25ad166738d82c547060ab778b64a9
I20260811 12:33:18.878795 6616 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33047
I20260811 12:33:18.879055 6616 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:18.879866 6616 heartbeater.cc:511] Master 127.3.204.126:33047 requested a full tablet report, sending...
I20260811 12:33:18.881497 5997 ts_manager.cc:194] Registered new tserver with Master: 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66:33493)
I20260811 12:33:18.882391 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:60103
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:18.919579 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33047
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:19.249038 6622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:19.249353 6622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:19.249495 6622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:19.257054 6622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:19.257181 6622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:19.257306 6622 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:33:19.257419 6622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:19.267664 6622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:33047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:19.269022 6622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:19.270351 6622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:19.283856 6634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:19.283895 6631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:19.283814 6638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:19.289095 6622 server_base.cc:1034] running on GCE node
I20260811 12:33:19.289842 6622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:19.290853 6622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:19.292100 6622 hybrid_clock.cc:648] HybridClock initialized: now 1786451599292006 us; error 44 us; skew 500 ppm
I20260811 12:33:19.292434 6622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:19 dist-test-slave-lgr5 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451599, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:19.295780 6622 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:19.297452 6622 webserver.cc:460] Webserver started at http://127.3.204.67:38279/ using document root <none> and password file <none>
I20260811 12:33:19.298200 6622 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:19.298288 6622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:19.298569 6622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:19.301802 6622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "661c2f22872348648ad0e2598bf40dc8"
format_stamp: "Formatted at 2026-08-11 12:33:19 on dist-test-slave-lgr5"
server_key: "24fc4a129df1f37f038ad5802c3b0358"
server_key_iv: "b35c33e12756ec017a961d6fcfcac27d"
server_key_version: "encryptionkey@0"
I20260811 12:33:19.302637 6622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "661c2f22872348648ad0e2598bf40dc8"
format_stamp: "Formatted at 2026-08-11 12:33:19 on dist-test-slave-lgr5"
server_key: "24fc4a129df1f37f038ad5802c3b0358"
server_key_iv: "b35c33e12756ec017a961d6fcfcac27d"
server_key_version: "encryptionkey@0"
I20260811 12:33:19.308142 6622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260811 12:33:19.311688 6648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:19.313136 6622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:19.313292 6622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "661c2f22872348648ad0e2598bf40dc8"
format_stamp: "Formatted at 2026-08-11 12:33:19 on dist-test-slave-lgr5"
server_key: "24fc4a129df1f37f038ad5802c3b0358"
server_key_iv: "b35c33e12756ec017a961d6fcfcac27d"
server_key_version: "encryptionkey@0"
I20260811 12:33:19.313534 6622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:19.325368 6622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:19.329833 6622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:19.330108 6622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:19.333303 6622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:19.333422 6622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:19.333593 6622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:19.333674 6622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:33:19 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451599, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:19.356693 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.333828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:48597 (local address 127.3.204.126:33047)
0811 12:33:19.334017 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.334024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.335788 (+ 1764us) server_negotiation.cc:407] Connection header received
0811 12:33:19.338019 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.338026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.338109 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.338199 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:19.339868 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.340933 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.343769 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.344434 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.344564 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.349247 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:19.349252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:19.349256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:19.349291 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:19.352132 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:19.353308 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:19.353314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:19.353317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:19.353483 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:19.354089 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:19.354093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:19.354095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:19.354461 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:19.354600 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.356045 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.356338 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 12:33:19.357798 6669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.334689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33047 (local address 127.3.204.67:48597)
0811 12:33:19.335559 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:19.335643 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:19.337713 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:19.338536 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:19.338545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:19.339071 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:19.339686 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.339705 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.341101 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.341111 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.343587 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.343606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.344877 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.344884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.345173 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.346010 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:19.346040 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:19.349026 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:19.352338 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:19.352349 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:19.352365 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:19.353131 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:19.353610 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:19.353616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:19.353620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:19.353915 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:19.354624 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:19.354632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:19.355467 (+ 835us) client_negotiation.cc:769] Sending connection context
0811 12:33:19.355911 (+ 444us) client_negotiation.cc:240] Negotiation successful
0811 12:33:19.356551 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":386,"threads_started":1}
I20260811 12:33:19.358466 6363 heartbeater.cc:503] Master 127.3.204.126:33047 was elected leader, sending a full tablet report...
I20260811 12:33:19.380030 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.364945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:41845 (local address 127.3.204.126:33047)
0811 12:33:19.365171 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.365179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.365691 (+ 512us) server_negotiation.cc:407] Connection header received
0811 12:33:19.365854 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.365860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.365973 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.366089 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:19.367016 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.368237 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.369654 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.370350 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.370482 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.373295 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:19.373301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:19.373304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:19.373347 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:19.376723 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:19.377692 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:19.377697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:19.377699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:19.377837 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:19.378308 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:19.378314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:19.378317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:19.378713 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:19.378893 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.379431 (+ 538us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.379653 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:33:19.380127 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.364918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33047 (local address 127.3.204.67:41845)
0811 12:33:19.365590 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:19.365608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:19.365740 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:19.366103 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:19.366106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:19.366569 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:19.366872 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.366882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.368413 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.368418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.369499 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.369514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.370570 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.370576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.370725 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.371488 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:19.371512 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:19.373082 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:19.376870 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:19.376880 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:19.376885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:19.377564 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:19.377963 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:19.377969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:19.377973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:19.378158 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:19.378867 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:19.378872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:19.379142 (+ 270us) client_negotiation.cc:769] Sending connection context
0811 12:33:19.379311 (+ 169us) client_negotiation.cc:240] Negotiation successful
0811 12:33:19.379678 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":343,"threads_started":1}
I20260811 12:33:19.393023 6622 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:45555
I20260811 12:33:19.393025 6844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:45555 every 8 connection(s)
I20260811 12:33:19.394323 6622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:19.396637 6846 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33047
I20260811 12:33:19.396878 6846 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:19.397543 6846 heartbeater.cc:511] Master 127.3.204.126:33047 requested a full tablet report, sending...
I20260811 12:33:19.398969 5997 ts_manager.cc:194] Registered new tserver with Master: 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555)
I20260811 12:33:19.399652 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 6622
I20260811 12:33:19.399827 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:19.399845 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:48597
I20260811 12:33:19.400419 3889 external_mini_cluster.cc:1428] Setting key 0ed66038b7dbd95529a0ffaa06112972
I20260811 12:33:19.403414 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 12:33:19 dist-test-slave-lgr5 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 12:33:19 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:19.451304 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.437296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.3.204.126:33047)
0811 12:33:19.437545 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.437552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.437731 (+ 179us) server_negotiation.cc:407] Connection header received
0811 12:33:19.437941 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.437945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.438015 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.438107 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:19.439567 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.440393 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.441724 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.442190 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.442309 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.446252 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:19.446257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:19.446260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:19.446299 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:19.448742 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:19.449367 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:19.449371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:19.449373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:19.449503 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:19.449871 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:19.449875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:19.449878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:19.450165 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:19.450416 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.450702 (+ 286us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.450939 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 12:33:19.455560 5997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:19.462970 5997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:19.490576 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.475912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38679 (local address 127.0.0.1:48898)
0811 12:33:19.481418 (+ 5506us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:19.481440 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:19.481564 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:19.482911 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:19.482919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:19.482950 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:19.483558 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.483567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.485838 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.485847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.488161 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.488171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.489497 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.489503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.489695 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.489748 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 12:33:19.489905 (+ 157us) client_negotiation.cc:240] Negotiation successful
0811 12:33:19.490106 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5381,"mutex_wait_us":63,"thread_start_us":3398,"threads_started":1}
I20260811 12:33:19.490667 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.475905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.3.204.67:45555)
0811 12:33:19.482953 (+ 7048us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.482961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.482992 (+ 31us) server_negotiation.cc:407] Connection header received
0811 12:33:19.483086 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.483093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.483281 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.483491 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:19.484524 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.485628 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.488134 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.489564 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.489698 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.489929 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.490167 (+ 238us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.490337 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6884,"thread_start_us":6680,"threads_started":1}
I20260811 12:33:19.491029 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.477012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.3.204.65:38679)
0811 12:33:19.482067 (+ 5055us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.482075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.482106 (+ 31us) server_negotiation.cc:407] Connection header received
0811 12:33:19.482214 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.482224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.482513 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.482727 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:19.483789 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.485421 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.488348 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.489292 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.489428 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.489663 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.490584 (+ 921us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.490762 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4894,"thread_start_us":4751,"threads_started":1}
I20260811 12:33:19.491163 6869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.477567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.3.204.66:33493)
0811 12:33:19.481781 (+ 4214us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.481791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.482333 (+ 542us) server_negotiation.cc:407] Connection header received
0811 12:33:19.482487 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.482495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.482735 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.483780 (+ 1045us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:19.484000 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.485421 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.488315 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.489601 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.489715 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.489900 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.490714 (+ 814us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.490889 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4040,"thread_start_us":3928,"threads_started":1}
I20260811 12:33:19.491341 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.475577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:45555 (local address 127.0.0.1:43102)
0811 12:33:19.476748 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:19.476793 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:19.476997 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:19.483776 (+ 6779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:19.483781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:19.483804 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:19.484280 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.484301 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.485752 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.485756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.487953 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.487966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.489817 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.489824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.489981 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.490012 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 12:33:19.490094 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 12:33:19.490214 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":67,"thread_start_us":5547,"threads_started":1}
I20260811 12:33:19.492065 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.476600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:33493 (local address 127.0.0.1:59338)
0811 12:33:19.482199 (+ 5599us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:19.482216 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:19.482338 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:19.482933 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:19.482936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:19.482972 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:19.483530 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.483561 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.485563 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.485569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.488138 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.488155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.490040 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.490045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.490227 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.490264 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 12:33:19.490361 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 12:33:19.490506 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5491,"mutex_wait_us":114,"spinlock_wait_cycles":2432,"thread_start_us":5352,"threads_started":1}
I20260811 12:33:19.495265 6234 tablet_service.cc:1467] Processing CreateTablet for tablet 323cc91084b44bfe8c05c0a2376de943 (DEFAULT_TABLE table=test-table [id=289a7027374d4bd4b4aaaa466c98dd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:19.495383 6739 tablet_service.cc:1467] Processing CreateTablet for tablet 323cc91084b44bfe8c05c0a2376de943 (DEFAULT_TABLE table=test-table [id=289a7027374d4bd4b4aaaa466c98dd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:19.495430 6468 tablet_service.cc:1467] Processing CreateTablet for tablet 323cc91084b44bfe8c05c0a2376de943 (DEFAULT_TABLE table=test-table [id=289a7027374d4bd4b4aaaa466c98dd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:19.496594 6234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323cc91084b44bfe8c05c0a2376de943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:19.496775 6468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323cc91084b44bfe8c05c0a2376de943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:19.497766 6739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323cc91084b44bfe8c05c0a2376de943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:19.507257 6870 tablet_bootstrap.cc:492] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Bootstrap starting.
I20260811 12:33:19.511400 6870 tablet_bootstrap.cc:654] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:19.511406 6872 tablet_bootstrap.cc:492] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Bootstrap starting.
I20260811 12:33:19.512919 6870 log.cc:824] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:19.514909 6874 tablet_bootstrap.cc:492] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Bootstrap starting.
I20260811 12:33:19.515705 6872 tablet_bootstrap.cc:654] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:19.516426 6870 tablet_bootstrap.cc:492] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: No bootstrap required, opened a new log
I20260811 12:33:19.516870 6870 ts_tablet_manager.cc:1397] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260811 12:33:19.517104 6872 log.cc:824] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:19.518800 6874 tablet_bootstrap.cc:654] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:19.520357 6874 log.cc:824] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:19.520876 6872 tablet_bootstrap.cc:492] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: No bootstrap required, opened a new log
I20260811 12:33:19.521281 6872 ts_tablet_manager.cc:1397] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260811 12:33:19.523119 6874 tablet_bootstrap.cc:492] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: No bootstrap required, opened a new log
I20260811 12:33:19.523351 6874 ts_tablet_manager.cc:1397] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 12:33:19.528966 6870 raft_consensus.cc:348] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.529655 6870 raft_consensus.cc:374] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:19.529804 6870 raft_consensus.cc:729] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3058bac42ec4f7286289fde5e1d43ce, State: Initialized, Role: FOLLOWER
I20260811 12:33:19.530529 6870 consensus_queue.cc:260] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.533166 6872 raft_consensus.cc:348] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.533655 6870 ts_tablet_manager.cc:1428] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260811 12:33:19.533597 6874 raft_consensus.cc:348] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.533983 6872 raft_consensus.cc:374] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:19.534271 6874 raft_consensus.cc:374] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:19.534206 6872 raft_consensus.cc:729] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afe16cda3a84d59b8c0c2d6b553ac13, State: Initialized, Role: FOLLOWER
I20260811 12:33:19.534550 6874 raft_consensus.cc:729] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661c2f22872348648ad0e2598bf40dc8, State: Initialized, Role: FOLLOWER
I20260811 12:33:19.535006 6872 consensus_queue.cc:260] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.535406 6874 consensus_queue.cc:260] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.536396 6616 heartbeater.cc:503] Master 127.3.204.126:33047 was elected leader, sending a full tablet report...
I20260811 12:33:19.536756 6846 heartbeater.cc:503] Master 127.3.204.126:33047 was elected leader, sending a full tablet report...
I20260811 12:33:19.536960 6872 ts_tablet_manager.cc:1428] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260811 12:33:19.537544 6874 ts_tablet_manager.cc:1428] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260811 12:33:19.610045 6364 tablet.cc:2369] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:33:19.627247 6618 tablet.cc:2369] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:33:19.637008 6882 raft_consensus.cc:482] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:19.637312 6882 raft_consensus.cc:504] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.639430 6882 leader_election.cc:288] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555), 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66:33493)
W20260811 12:33:19.646026 6847 tablet.cc:2369] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:33:19.647215 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.640068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:58941 (local address 127.3.204.67:45555)
0811 12:33:19.640508 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.640514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.640922 (+ 408us) server_negotiation.cc:407] Connection header received
0811 12:33:19.641117 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.641122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.641180 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.641253 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:19.641878 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.642667 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.645224 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.646186 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.646362 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.646440 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.646863 (+ 423us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.646993 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 12:33:19.647430 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.639800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:45555 (local address 127.3.204.65:58941)
0811 12:33:19.640787 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:19.640816 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:19.640973 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:19.641280 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:19.641285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:19.641308 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:19.641737 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.641747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.642914 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.642920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.645052 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.645066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.646408 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.646416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.646671 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.646711 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 12:33:19.646814 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 12:33:19.646967 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":20,"spinlock_wait_cycles":5888,"thread_start_us":698,"threads_started":1}
I20260811 12:33:19.648252 6869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.639910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:41575 (local address 127.3.204.66:33493)
0811 12:33:19.640297 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.640305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.641015 (+ 710us) server_negotiation.cc:407] Connection header received
0811 12:33:19.641195 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.641199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.641291 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.641375 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:19.642395 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.643411 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.645754 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.647187 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.647408 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.647489 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.647835 (+ 346us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.647978 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260811 12:33:19.648442 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.639935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:33493 (local address 127.3.204.65:41575)
0811 12:33:19.640916 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:19.640940 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:19.641069 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:19.641417 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:19.641422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:19.641482 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:19.642194 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.642205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.643601 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.643606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.645569 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:19.645582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.647444 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.647461 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:19.647628 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.647661 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 12:33:19.647797 (+ 136us) client_negotiation.cc:240] Negotiation successful
0811 12:33:19.647927 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":666,"threads_started":1}
I20260811 12:33:19.648808 6774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323cc91084b44bfe8c05c0a2376de943" candidate_uuid: "b3058bac42ec4f7286289fde5e1d43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661c2f22872348648ad0e2598bf40dc8" is_pre_election: true
I20260811 12:33:19.649729 6774 raft_consensus.cc:2454] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3058bac42ec4f7286289fde5e1d43ce in term 0.
I20260811 12:33:19.649624 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323cc91084b44bfe8c05c0a2376de943" candidate_uuid: "b3058bac42ec4f7286289fde5e1d43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" is_pre_election: true
I20260811 12:33:19.650285 6507 raft_consensus.cc:2454] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3058bac42ec4f7286289fde5e1d43ce in term 0.
I20260811 12:33:19.650622 6142 leader_election.cc:302] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661c2f22872348648ad0e2598bf40dc8, b3058bac42ec4f7286289fde5e1d43ce; no voters:
I20260811 12:33:19.651055 6882 raft_consensus.cc:2781] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:19.651197 6882 raft_consensus.cc:482] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:19.651295 6882 raft_consensus.cc:3037] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:19.654088 6882 raft_consensus.cc:504] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.654866 6882 leader_election.cc:288] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [CANDIDATE]: Term 1 election: Requested vote from peers 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555), 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66:33493)
I20260811 12:33:19.655370 6774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323cc91084b44bfe8c05c0a2376de943" candidate_uuid: "b3058bac42ec4f7286289fde5e1d43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661c2f22872348648ad0e2598bf40dc8"
I20260811 12:33:19.655370 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323cc91084b44bfe8c05c0a2376de943" candidate_uuid: "b3058bac42ec4f7286289fde5e1d43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13"
I20260811 12:33:19.655596 6774 raft_consensus.cc:3037] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:19.655594 6507 raft_consensus.cc:3037] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:19.659224 6774 raft_consensus.cc:2454] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3058bac42ec4f7286289fde5e1d43ce in term 1.
I20260811 12:33:19.659250 6507 raft_consensus.cc:2454] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3058bac42ec4f7286289fde5e1d43ce in term 1.
I20260811 12:33:19.659847 6145 leader_election.cc:302] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5afe16cda3a84d59b8c0c2d6b553ac13, 661c2f22872348648ad0e2598bf40dc8, b3058bac42ec4f7286289fde5e1d43ce; no voters:
I20260811 12:33:19.660341 6882 raft_consensus.cc:2781] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:19.660851 6882 raft_consensus.cc:686] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 1 LEADER]: Becoming Leader. State: Replica: b3058bac42ec4f7286289fde5e1d43ce, State: Running, Role: LEADER
I20260811 12:33:19.661430 6882 consensus_queue.cc:237] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } }
I20260811 12:33:19.666530 5995 catalog_manager.cc:5495] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce reported cstate change: term changed from 0 to 1, leader changed from <none> to b3058bac42ec4f7286289fde5e1d43ce (127.3.204.65). New cstate: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } health_report { overall_health: UNKNOWN } } }
I20260811 12:33:19.708705 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:19.699162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.3.204.65:38679)
0811 12:33:19.699450 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:19.699457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:19.699482 (+ 25us) server_negotiation.cc:407] Connection header received
0811 12:33:19.699854 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:19.699860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:19.699972 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:19.700135 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:19.700862 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.702368 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.705035 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:19.705779 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:19.705914 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:19.706730 (+ 816us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:19.707571 (+ 841us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:19.707707 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:19.708227 (+ 520us) server_negotiation.cc:299] Negotiation successful
0811 12:33:19.708329 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 12:33:19.720868 6507 raft_consensus.cc:1261] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Refusing update from remote peer b3058bac42ec4f7286289fde5e1d43ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:19.720882 6774 raft_consensus.cc:1261] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Refusing update from remote peer b3058bac42ec4f7286289fde5e1d43ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:19.722203 6891 consensus_queue.cc:1035] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:19.723712 6891 consensus_queue.cc:1035] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:19.739563 6899 mvcc.cc:205] Tried to move back new op lower bound from 7317305752444489728 to 7317305752216993792. Current Snapshot: MvccSnapshot[applied={T|T < 7317305752444489728}]
I20260811 12:33:19.770488 6000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:19.770965 6000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:19.775341 6000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8: Sending DeleteTablet for 3 replicas of tablet 323cc91084b44bfe8c05c0a2376de943
I20260811 12:33:19.776878 6739 tablet_service.cc:1514] Processing DeleteTablet for tablet 323cc91084b44bfe8c05c0a2376de943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:19 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:43102
I20260811 12:33:19.776926 6234 tablet_service.cc:1514] Processing DeleteTablet for tablet 323cc91084b44bfe8c05c0a2376de943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:19 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:48898
I20260811 12:33:19.778214 6916 tablet_replica.cc:330] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: stopping tablet replica
I20260811 12:33:19.778219 6468 tablet_service.cc:1514] Processing DeleteTablet for tablet 323cc91084b44bfe8c05c0a2376de943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:19 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:59338
I20260811 12:33:19.778898 6916 raft_consensus.cc:2229] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:19.779839 6916 raft_consensus.cc:2258] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:19.780000 6000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:19.780860 6000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:19.780876 6917 tablet_replica.cc:330] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: stopping tablet replica
I20260811 12:33:19.781337 6917 raft_consensus.cc:2229] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:33:19.782229 6917 raft_consensus.cc:2258] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:19.789124 6917 ts_tablet_manager.cc:1905] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:19.789649 6465 tablet_service.cc:1467] Processing CreateTablet for tablet 0cfdca07d5254d2c8a125a065a476849 (DEFAULT_TABLE table=test-table [id=04eaca9f612449e792388f2b8520926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:19.789856 6234 tablet_service.cc:1467] Processing CreateTablet for tablet 0cfdca07d5254d2c8a125a065a476849 (DEFAULT_TABLE table=test-table [id=04eaca9f612449e792388f2b8520926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:19.790256 6465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cfdca07d5254d2c8a125a065a476849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:19.790575 6234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cfdca07d5254d2c8a125a065a476849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:19.794209 6739 tablet_service.cc:1467] Processing CreateTablet for tablet 0cfdca07d5254d2c8a125a065a476849 (DEFAULT_TABLE table=test-table [id=04eaca9f612449e792388f2b8520926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:19.794690 6739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cfdca07d5254d2c8a125a065a476849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:19.797607 6916 ts_tablet_manager.cc:1905] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:19.798508 6917 ts_tablet_manager.cc:1918] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:33:19.798802 6917 log.cc:1196] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/323cc91084b44bfe8c05c0a2376de943
I20260811 12:33:19.799639 6917 ts_tablet_manager.cc:1939] T 323cc91084b44bfe8c05c0a2376de943 P b3058bac42ec4f7286289fde5e1d43ce: Deleting consensus metadata
I20260811 12:33:19.801550 6870 tablet_bootstrap.cc:492] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: Bootstrap starting.
I20260811 12:33:19.802460 5970 catalog_manager.cc:4841] TS b3058bac42ec4f7286289fde5e1d43ce (127.3.204.65:38679): tablet 323cc91084b44bfe8c05c0a2376de943 (table test-table [id=289a7027374d4bd4b4aaaa466c98dd6b]) successfully deleted
I20260811 12:33:19.802930 6870 tablet_bootstrap.cc:654] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:19.805711 6874 tablet_bootstrap.cc:492] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: Bootstrap starting.
I20260811 12:33:19.807590 6918 tablet_replica.cc:330] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: stopping tablet replica
I20260811 12:33:19.807700 6872 tablet_bootstrap.cc:492] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: Bootstrap starting.
I20260811 12:33:19.807899 6874 tablet_bootstrap.cc:654] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:19.808122 6918 raft_consensus.cc:2229] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:19.809011 6870 tablet_bootstrap.cc:492] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: No bootstrap required, opened a new log
I20260811 12:33:19.809149 6870 ts_tablet_manager.cc:1397] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260811 12:33:19.809928 6870 raft_consensus.cc:348] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:19.810123 6870 raft_consensus.cc:374] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:19.810250 6870 raft_consensus.cc:729] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3058bac42ec4f7286289fde5e1d43ce, State: Initialized, Role: FOLLOWER
I20260811 12:33:19.810626 6870 consensus_queue.cc:260] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:19.811852 6870 ts_tablet_manager.cc:1428] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260811 12:33:19.812973 6872 tablet_bootstrap.cc:654] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:19.813104 6918 raft_consensus.cc:2258] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:19.817072 6916 ts_tablet_manager.cc:1918] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:33:19.817272 6916 log.cc:1196] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/323cc91084b44bfe8c05c0a2376de943
I20260811 12:33:19.818274 6916 ts_tablet_manager.cc:1939] T 323cc91084b44bfe8c05c0a2376de943 P 661c2f22872348648ad0e2598bf40dc8: Deleting consensus metadata
I20260811 12:33:19.819285 6918 ts_tablet_manager.cc:1905] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:19.820235 5969 catalog_manager.cc:4841] TS 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555): tablet 323cc91084b44bfe8c05c0a2376de943 (table test-table [id=289a7027374d4bd4b4aaaa466c98dd6b]) successfully deleted
I20260811 12:33:19.821910 6874 tablet_bootstrap.cc:492] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: No bootstrap required, opened a new log
I20260811 12:33:19.822078 6874 ts_tablet_manager.cc:1397] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.011s
I20260811 12:33:19.822908 6872 tablet_bootstrap.cc:492] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: No bootstrap required, opened a new log
I20260811 12:33:19.823112 6872 ts_tablet_manager.cc:1397] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20260811 12:33:19.822997 6874 raft_consensus.cc:348] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:19.823205 6874 raft_consensus.cc:374] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:19.823357 6874 raft_consensus.cc:729] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661c2f22872348648ad0e2598bf40dc8, State: Initialized, Role: FOLLOWER
I20260811 12:33:19.823829 6874 consensus_queue.cc:260] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:19.824003 6872 raft_consensus.cc:348] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:19.824360 6872 raft_consensus.cc:374] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:19.824522 6872 raft_consensus.cc:729] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afe16cda3a84d59b8c0c2d6b553ac13, State: Initialized, Role: FOLLOWER
I20260811 12:33:19.824798 6872 consensus_queue.cc:260] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:19.827185 6874 ts_tablet_manager.cc:1428] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260811 12:33:19.829823 6872 ts_tablet_manager.cc:1428] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260811 12:33:19.834362 6918 ts_tablet_manager.cc:1918] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:33:19.834565 6918 log.cc:1196] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/323cc91084b44bfe8c05c0a2376de943
I20260811 12:33:19.835223 6918 ts_tablet_manager.cc:1939] T 323cc91084b44bfe8c05c0a2376de943 P 5afe16cda3a84d59b8c0c2d6b553ac13: Deleting consensus metadata
I20260811 12:33:19.837301 5971 catalog_manager.cc:4841] TS 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66:33493): tablet 323cc91084b44bfe8c05c0a2376de943 (table test-table [id=289a7027374d4bd4b4aaaa466c98dd6b]) successfully deleted
I20260811 12:33:20.263494 6886 raft_consensus.cc:482] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:20.263741 6886 raft_consensus.cc:504] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.265684 6886 leader_election.cc:288] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3058bac42ec4f7286289fde5e1d43ce (127.3.204.65:38679), 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555)
I20260811 12:33:20.280881 6887 raft_consensus.cc:482] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:20.281136 6887 raft_consensus.cc:504] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.282963 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.266152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:56727 (local address 127.3.204.67:45555)
0811 12:33:20.277405 (+ 11253us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.277411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.277432 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:33:20.277512 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.277518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.277582 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.277673 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:20.278244 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.278955 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.280846 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.281982 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.282146 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.282218 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.282522 (+ 304us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.282670 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11133,"thread_start_us":11033,"threads_started":1}
I20260811 12:33:20.283016 6887 leader_election.cc:288] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3058bac42ec4f7286289fde5e1d43ce (127.3.204.65:38679), 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66:33493)
I20260811 12:33:20.283226 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.265929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:45555 (local address 127.3.204.66:56727)
0811 12:33:20.276835 (+ 10906us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:20.276865 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:20.277078 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:20.277714 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:20.277719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:20.277747 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:20.278089 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.278101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.279100 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.279104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.280640 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.280654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.282151 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.282157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.282342 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.282379 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 12:33:20.282480 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 12:33:20.282625 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10748,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":10965,"threads_started":1}
I20260811 12:33:20.283776 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.265929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38679 (local address 127.3.204.66:55115)
0811 12:33:20.277217 (+ 11288us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:20.277238 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:20.277405 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:20.277835 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:20.277840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:20.277915 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:20.278276 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.278286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.279281 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.279286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.281292 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.281305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.282894 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.282900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.283086 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.283120 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 12:33:20.283197 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 12:33:20.283324 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11160,"mutex_wait_us":491,"spinlock_wait_cycles":896,"thread_start_us":10104,"threads_started":1}
I20260811 12:33:20.284538 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.266250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:55115 (local address 127.3.204.65:38679)
0811 12:33:20.276204 (+ 9954us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.276211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.277358 (+ 1147us) server_negotiation.cc:407] Connection header received
0811 12:33:20.277591 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.277596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.277661 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.277769 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:20.278427 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.279146 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.281490 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.282540 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.282680 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.282751 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.283999 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.284182 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9768,"thread_start_us":9661,"threads_started":1}
I20260811 12:33:20.286113 6774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfdca07d5254d2c8a125a065a476849" candidate_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661c2f22872348648ad0e2598bf40dc8" is_pre_election: true
I20260811 12:33:20.286921 6774 raft_consensus.cc:2454] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5afe16cda3a84d59b8c0c2d6b553ac13 in term 0.
I20260811 12:33:20.287272 6269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfdca07d5254d2c8a125a065a476849" candidate_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3058bac42ec4f7286289fde5e1d43ce" is_pre_election: true
I20260811 12:33:20.287545 6395 leader_election.cc:302] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5afe16cda3a84d59b8c0c2d6b553ac13, 661c2f22872348648ad0e2598bf40dc8; no voters:
I20260811 12:33:20.287736 6269 raft_consensus.cc:2454] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5afe16cda3a84d59b8c0c2d6b553ac13 in term 0.
I20260811 12:33:20.287990 6886 raft_consensus.cc:2781] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:20.288123 6886 raft_consensus.cc:482] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:20.288209 6886 raft_consensus.cc:3037] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:20.291680 6886 raft_consensus.cc:504] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.292616 6886 leader_election.cc:288] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [CANDIDATE]: Term 1 election: Requested vote from peers b3058bac42ec4f7286289fde5e1d43ce (127.3.204.65:38679), 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555)
I20260811 12:33:20.292945 6269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfdca07d5254d2c8a125a065a476849" candidate_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3058bac42ec4f7286289fde5e1d43ce"
I20260811 12:33:20.293159 6269 raft_consensus.cc:3037] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:20.293576 6774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfdca07d5254d2c8a125a065a476849" candidate_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661c2f22872348648ad0e2598bf40dc8"
I20260811 12:33:20.293877 6774 raft_consensus.cc:3037] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:20.294061 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.283805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:40321 (local address 127.3.204.65:38679)
0811 12:33:20.284786 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.284792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.285523 (+ 731us) server_negotiation.cc:407] Connection header received
0811 12:33:20.285787 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.285793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.285879 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.285944 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:20.287170 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.288264 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.291385 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.292696 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.292867 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.292944 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.293656 (+ 712us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.293784 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":2704,"threads_started":1}
I20260811 12:33:20.294747 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.283431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38679 (local address 127.3.204.67:40321)
0811 12:33:20.285437 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:20.285456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:20.285637 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:20.286139 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:20.286144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:20.286173 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:20.286983 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.287000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.288472 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.288481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.291173 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.291189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.293062 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.293068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.293231 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.293268 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 12:33:20.293371 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 12:33:20.293506 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1871,"mutex_wait_us":3,"thread_start_us":3089,"threads_started":1}
I20260811 12:33:20.295678 6267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfdca07d5254d2c8a125a065a476849" candidate_uuid: "661c2f22872348648ad0e2598bf40dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3058bac42ec4f7286289fde5e1d43ce" is_pre_election: true
I20260811 12:33:20.296335 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.283838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:35259 (local address 127.3.204.66:33493)
0811 12:33:20.285345 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.285352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.286907 (+ 1555us) server_negotiation.cc:407] Connection header received
0811 12:33:20.287476 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.287481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.287581 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.287652 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:20.288998 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.290322 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.292312 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.294333 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.294645 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.294725 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.295860 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.296003 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":1176,"threads_started":1}
I20260811 12:33:20.296428 6269 raft_consensus.cc:2454] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5afe16cda3a84d59b8c0c2d6b553ac13 in term 1.
I20260811 12:33:20.297066 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.283431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:33493 (local address 127.3.204.67:35259)
0811 12:33:20.286811 (+ 3380us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:20.286827 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:20.287288 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:20.288368 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:20.288372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:20.288400 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:20.288806 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.288815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.290455 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.290459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.292123 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.292138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.295400 (+ 3262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.295407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.295665 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.295701 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:33:20.296512 (+ 811us) client_negotiation.cc:240] Negotiation successful
0811 12:33:20.296656 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3286,"mutex_wait_us":270,"thread_start_us":1499,"threads_started":1}
I20260811 12:33:20.297264 6396 leader_election.cc:302] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5afe16cda3a84d59b8c0c2d6b553ac13, b3058bac42ec4f7286289fde5e1d43ce; no voters:
I20260811 12:33:20.297693 6886 raft_consensus.cc:2781] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:20.298035 6774 raft_consensus.cc:2454] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5afe16cda3a84d59b8c0c2d6b553ac13 in term 1.
I20260811 12:33:20.298095 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfdca07d5254d2c8a125a065a476849" candidate_uuid: "661c2f22872348648ad0e2598bf40dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" is_pre_election: true
I20260811 12:33:20.298653 6886 raft_consensus.cc:686] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 LEADER]: Becoming Leader. State: Replica: 5afe16cda3a84d59b8c0c2d6b553ac13, State: Running, Role: LEADER
I20260811 12:33:20.299057 6886 consensus_queue.cc:237] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.301702 6654 leader_election.cc:302] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 661c2f22872348648ad0e2598bf40dc8; no voters: 5afe16cda3a84d59b8c0c2d6b553ac13, b3058bac42ec4f7286289fde5e1d43ce
I20260811 12:33:20.302155 6887 raft_consensus.cc:2726] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 12:33:20.303028 5995 catalog_manager.cc:5495] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66). New cstate: current_term: 1 leader_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } health_report { overall_health: UNKNOWN } } }
Aug 11 12:33:20 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:20.412310 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.394902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33047 (local address 127.0.0.1:44890)
0811 12:33:20.396328 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:20.396346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:20.396542 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:20.396890 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:20.396895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:20.397682 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:20.398160 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.398176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.399456 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.399469 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.400572 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.400585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.401505 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.401512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.401677 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.402495 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:20.402514 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:20.405484 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:20.409262 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:20.409275 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:20.409284 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:20.409789 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:20.410162 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:20.410167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:20.410171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:20.410423 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:20.410971 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:20.410980 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:20.411208 (+ 228us) client_negotiation.cc:769] Sending connection context
0811 12:33:20.411414 (+ 206us) client_negotiation.cc:240] Negotiation successful
0811 12:33:20.411717 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"thread_start_us":1042,"threads_started":1}
I20260811 12:33:20.412308 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.394888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.3.204.126:33047)
0811 12:33:20.395910 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.395917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.396439 (+ 522us) server_negotiation.cc:407] Connection header received
0811 12:33:20.396697 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.396701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.396759 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.396850 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:20.398312 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.399327 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.400713 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.401332 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.401447 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.405745 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:20.405752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:20.405756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:20.405788 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:20.409138 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.409911 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.409915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.409918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.410057 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.410542 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.410545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.410547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.410834 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:20.410978 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.411553 (+ 575us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.411854 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":748,"thread_start_us":354,"threads_started":1}
I20260811 12:33:20.418205 5995 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:44890:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 12:33:20.419562 5995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:20.426558 6234 tablet_service.cc:1467] Processing CreateTablet for tablet 8daed59346c541fcbb6ad8f287b46e51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5449ea46c18431fa386bb4db4772228]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:33:20.426717 6465 tablet_service.cc:1467] Processing CreateTablet for tablet 8daed59346c541fcbb6ad8f287b46e51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5449ea46c18431fa386bb4db4772228]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:33:20.426749 6739 tablet_service.cc:1467] Processing CreateTablet for tablet 8daed59346c541fcbb6ad8f287b46e51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5449ea46c18431fa386bb4db4772228]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:33:20.427219 6234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8daed59346c541fcbb6ad8f287b46e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:20.427310 6739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8daed59346c541fcbb6ad8f287b46e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:20.427354 6465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8daed59346c541fcbb6ad8f287b46e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:20.434464 6970 tablet_bootstrap.cc:492] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: Bootstrap starting.
I20260811 12:33:20.435169 6973 tablet_bootstrap.cc:492] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: Bootstrap starting.
I20260811 12:33:20.435930 6971 tablet_bootstrap.cc:492] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: Bootstrap starting.
I20260811 12:33:20.437013 6973 tablet_bootstrap.cc:654] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:20.438078 6970 tablet_bootstrap.cc:654] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:20.438405 6971 tablet_bootstrap.cc:654] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:20.442704 6973 tablet_bootstrap.cc:492] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: No bootstrap required, opened a new log
I20260811 12:33:20.442881 6970 tablet_bootstrap.cc:492] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: No bootstrap required, opened a new log
I20260811 12:33:20.443007 6973 ts_tablet_manager.cc:1397] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260811 12:33:20.443140 6970 ts_tablet_manager.cc:1397] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260811 12:33:20.443529 6971 tablet_bootstrap.cc:492] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: No bootstrap required, opened a new log
I20260811 12:33:20.443806 6971 ts_tablet_manager.cc:1397] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260811 12:33:20.444275 6973 raft_consensus.cc:348] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.444676 6973 raft_consensus.cc:374] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:20.444577 6970 raft_consensus.cc:348] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.444904 6970 raft_consensus.cc:374] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:20.444917 6973 raft_consensus.cc:729] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afe16cda3a84d59b8c0c2d6b553ac13, State: Initialized, Role: FOLLOWER
I20260811 12:33:20.445102 6970 raft_consensus.cc:729] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661c2f22872348648ad0e2598bf40dc8, State: Initialized, Role: FOLLOWER
I20260811 12:33:20.445406 6973 consensus_queue.cc:260] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.445396 6971 raft_consensus.cc:348] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.445654 6971 raft_consensus.cc:374] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:20.445729 6971 raft_consensus.cc:729] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3058bac42ec4f7286289fde5e1d43ce, State: Initialized, Role: FOLLOWER
I20260811 12:33:20.445770 6970 consensus_queue.cc:260] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.445925 6886 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.445997 6971 consensus_queue.cc:260] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.446235 6886 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:20.446417 6887 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.446897 6887 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:20.448181 6970 ts_tablet_manager.cc:1428] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260811 12:33:20.449003 6971 ts_tablet_manager.cc:1428] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260811 12:33:20.449258 6973 ts_tablet_manager.cc:1428] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260811 12:33:20.449697 6981 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.450192 6981 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:20.513134 6981 raft_consensus.cc:482] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:20.513332 6981 raft_consensus.cc:504] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.514014 6981 leader_election.cc:288] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66:33493), 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555)
I20260811 12:33:20.514638 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8daed59346c541fcbb6ad8f287b46e51" candidate_uuid: "b3058bac42ec4f7286289fde5e1d43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" is_pre_election: true
I20260811 12:33:20.514714 6774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8daed59346c541fcbb6ad8f287b46e51" candidate_uuid: "b3058bac42ec4f7286289fde5e1d43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661c2f22872348648ad0e2598bf40dc8" is_pre_election: true
I20260811 12:33:20.514892 6507 raft_consensus.cc:2454] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3058bac42ec4f7286289fde5e1d43ce in term 0.
I20260811 12:33:20.515013 6774 raft_consensus.cc:2454] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3058bac42ec4f7286289fde5e1d43ce in term 0.
I20260811 12:33:20.515455 6142 leader_election.cc:302] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5afe16cda3a84d59b8c0c2d6b553ac13, 661c2f22872348648ad0e2598bf40dc8, b3058bac42ec4f7286289fde5e1d43ce; no voters:
I20260811 12:33:20.515796 6981 raft_consensus.cc:2781] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:20.516012 6981 raft_consensus.cc:482] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:20.516211 6981 raft_consensus.cc:3037] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:20.519567 6981 raft_consensus.cc:504] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.520443 6981 leader_election.cc:288] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [CANDIDATE]: Term 1 election: Requested vote from peers 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66:33493), 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555)
I20260811 12:33:20.520982 6774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8daed59346c541fcbb6ad8f287b46e51" candidate_uuid: "b3058bac42ec4f7286289fde5e1d43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661c2f22872348648ad0e2598bf40dc8"
I20260811 12:33:20.521078 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8daed59346c541fcbb6ad8f287b46e51" candidate_uuid: "b3058bac42ec4f7286289fde5e1d43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13"
I20260811 12:33:20.521241 6774 raft_consensus.cc:3037] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:20.521381 6507 raft_consensus.cc:3037] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:20.524777 6774 raft_consensus.cc:2454] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3058bac42ec4f7286289fde5e1d43ce in term 1.
I20260811 12:33:20.524777 6507 raft_consensus.cc:2454] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3058bac42ec4f7286289fde5e1d43ce in term 1.
I20260811 12:33:20.525311 6142 leader_election.cc:302] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5afe16cda3a84d59b8c0c2d6b553ac13, 661c2f22872348648ad0e2598bf40dc8, b3058bac42ec4f7286289fde5e1d43ce; no voters:
I20260811 12:33:20.525700 6981 raft_consensus.cc:2781] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:20.526316 6981 raft_consensus.cc:686] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [term 1 LEADER]: Becoming Leader. State: Replica: b3058bac42ec4f7286289fde5e1d43ce, State: Running, Role: LEADER
I20260811 12:33:20.526682 6981 consensus_queue.cc:237] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } }
I20260811 12:33:20.528949 6982 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: TxnStatusTablet state changed. Reason: New leader b3058bac42ec4f7286289fde5e1d43ce. Latest consensus state: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.529357 6982 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:33:20.530308 6986 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 12:33:20.530305 6000 catalog_manager.cc:5495] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce reported cstate change: term changed from 0 to 1, leader changed from <none> to b3058bac42ec4f7286289fde5e1d43ce (127.3.204.65). New cstate: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } health_report { overall_health: UNKNOWN } } }
I20260811 12:33:20.530664 6986 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 12:33:20.567984 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.561867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38679 (local address 127.0.0.1:35556)
0811 12:33:20.562187 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:20.562210 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:20.562391 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:20.562818 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:20.562822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:20.562835 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:20.563101 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.563112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.564603 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.564607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.565690 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.565703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.566687 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.566694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.566886 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.566952 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:20.567390 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:20.567414 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 12:33:20.567486 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 12:33:20.567652 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 12:33:20.568104 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.562030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.3.204.65:38679)
0811 12:33:20.562223 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.562230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.562327 (+ 97us) server_negotiation.cc:407] Connection header received
0811 12:33:20.562585 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.562589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.562685 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.562759 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:20.563242 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.564476 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.565831 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.566530 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.566648 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.567167 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:20.567291 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:20.567368 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.567525 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.567608 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 12:33:20.574003 6774 raft_consensus.cc:1261] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Refusing update from remote peer b3058bac42ec4f7286289fde5e1d43ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:20.574047 6507 raft_consensus.cc:1261] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Refusing update from remote peer b3058bac42ec4f7286289fde5e1d43ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:20.574716 6981 consensus_queue.cc:1035] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:20.575129 6982 consensus_queue.cc:1035] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:20.585103 6886 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: TxnStatusTablet state changed. Reason: New leader b3058bac42ec4f7286289fde5e1d43ce. Latest consensus state: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.585470 6886 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:20.587744 6887 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: TxnStatusTablet state changed. Reason: New leader b3058bac42ec4f7286289fde5e1d43ce. Latest consensus state: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.588137 6887 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:20.590763 6982 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.591109 6982 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:33:20.592864 6989 mvcc.cc:205] Tried to move back new op lower bound from 7317305755942260736 to 7317305755759718400. Current Snapshot: MvccSnapshot[applied={T|T < 7317305755942260736 or (T in {7317305755942260736})}]
I20260811 12:33:20.625211 6981 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.625777 6981 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P b3058bac42ec4f7286289fde5e1d43ce: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:33:20.629164 6991 mvcc.cc:205] Tried to move back new op lower bound from 7317305755942260736 to 7317305755759718400. Current Snapshot: MvccSnapshot[applied={T|T < 7317305755942260736}]
I20260811 12:33:20.635188 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.611348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.3.204.66:33493)
0811 12:33:20.611632 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.611639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.621905 (+ 10266us) server_negotiation.cc:407] Connection header received
0811 12:33:20.622585 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.622593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.622712 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.622856 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:20.624962 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.626300 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.631359 (+ 5059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.632280 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.632388 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.633238 (+ 850us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:20.634116 (+ 878us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:20.634305 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.634600 (+ 295us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.634728 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:33:20.640569 6887 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.640851 6887 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P 661c2f22872348648ad0e2598bf40dc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:20.641187 6886 tablet_replica.cc:439] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3058bac42ec4f7286289fde5e1d43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 } } peers { permanent_uuid: "5afe16cda3a84d59b8c0c2d6b553ac13" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 33493 } } peers { permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 } } }
I20260811 12:33:20.641548 6886 tablet_replica.cc:442] T 8daed59346c541fcbb6ad8f287b46e51 P 5afe16cda3a84d59b8c0c2d6b553ac13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:20.658360 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.651599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38679 (local address 127.3.204.66:57123)
0811 12:33:20.652457 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:20.652477 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:20.652623 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:20.652948 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:20.652953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:20.652965 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:20.653264 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.653275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.654832 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.654836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.656250 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.656264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.657167 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.657174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.657298 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.657357 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:20.657740 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:20.657752 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 12:33:20.657828 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 12:33:20.657971 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":601,"threads_started":1}
I20260811 12:33:20.658455 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.651909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:57123 (local address 127.3.204.65:38679)
0811 12:33:20.652161 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.652167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.652565 (+ 398us) server_negotiation.cc:407] Connection header received
0811 12:33:20.652762 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.652767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.652844 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.652917 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:20.653411 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.654658 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.656389 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.656966 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.657103 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.657528 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:20.657647 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:20.657719 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.657860 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.657945 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 12:33:20.671463 6774 raft_consensus.cc:1261] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Refusing update from remote peer 5afe16cda3a84d59b8c0c2d6b553ac13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:20.671800 6269 raft_consensus.cc:1261] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Refusing update from remote peer 5afe16cda3a84d59b8c0c2d6b553ac13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:20.672360 6946 consensus_queue.cc:1035] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661c2f22872348648ad0e2598bf40dc8" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:20.672863 6886 consensus_queue.cc:1035] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3058bac42ec4f7286289fde5e1d43ce" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:20.730316 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.712018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:47349 (local address 127.3.204.66:33493)
0811 12:33:20.713020 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.713027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.722397 (+ 9370us) server_negotiation.cc:407] Connection header received
0811 12:33:20.723132 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.723140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.723256 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.723327 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:20.724473 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.725759 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.727989 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.728594 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.728725 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.729477 (+ 752us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:20.729628 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:20.729691 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.729908 (+ 217us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.729986 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843}
I20260811 12:33:20.730810 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.712146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:33493 (local address 127.3.204.65:47349)
0811 12:33:20.722275 (+ 10129us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:20.722300 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:20.722504 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:20.723596 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:20.723601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:20.723614 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:20.724272 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.724288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.725917 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.725921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.727844 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:20.727857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.728798 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.728806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:20.729116 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.729249 (+ 133us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:20.729773 (+ 524us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:20.729784 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 12:33:20.729877 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 12:33:20.730031 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9823,"thread_start_us":10485,"threads_started":1}
I20260811 12:33:20.779984 6000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:20.780300 6000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:20.784400 6000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 188cc6c25ddf4a2cacf47ffffb3f0cc8: Sending DeleteTablet for 3 replicas of tablet 0cfdca07d5254d2c8a125a065a476849
I20260811 12:33:20.785362 6234 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cfdca07d5254d2c8a125a065a476849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:20 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:48898
I20260811 12:33:20.785769 6739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cfdca07d5254d2c8a125a065a476849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:20 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:43102
I20260811 12:33:20.786504 6465 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cfdca07d5254d2c8a125a065a476849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:20 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:59338
I20260811 12:33:20.796212 7021 tablet_replica.cc:330] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: stopping tablet replica
I20260811 12:33:20.796456 7021 raft_consensus.cc:2229] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:20.796725 7021 raft_consensus.cc:2258] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:20.797921 7021 ts_tablet_manager.cc:1905] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:20.805830 7021 ts_tablet_manager.cc:1918] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:20.805989 7021 log.cc:1196] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/0cfdca07d5254d2c8a125a065a476849
I20260811 12:33:20.806452 7021 ts_tablet_manager.cc:1939] T 0cfdca07d5254d2c8a125a065a476849 P 661c2f22872348648ad0e2598bf40dc8: Deleting consensus metadata
I20260811 12:33:20.807700 5969 catalog_manager.cc:4841] TS 661c2f22872348648ad0e2598bf40dc8 (127.3.204.67:45555): tablet 0cfdca07d5254d2c8a125a065a476849 (table test-table [id=04eaca9f612449e792388f2b8520926e]) successfully deleted
I20260811 12:33:20.808455 7020 tablet_replica.cc:330] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: stopping tablet replica
I20260811 12:33:20.808702 7020 raft_consensus.cc:2229] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:20.809060 7022 tablet_replica.cc:330] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: stopping tablet replica
I20260811 12:33:20.809146 7020 raft_consensus.cc:2258] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:20.810504 7020 ts_tablet_manager.cc:1905] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:20.818657 7020 ts_tablet_manager.cc:1918] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:20.818856 7022 raft_consensus.cc:2229] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:33:20.819983 7022 raft_consensus.cc:2258] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:20.821542 7022 ts_tablet_manager.cc:1905] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:20.824133 7020 log.cc:1196] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/0cfdca07d5254d2c8a125a065a476849
I20260811 12:33:20.824608 7020 ts_tablet_manager.cc:1939] T 0cfdca07d5254d2c8a125a065a476849 P b3058bac42ec4f7286289fde5e1d43ce: Deleting consensus metadata
I20260811 12:33:20.825762 5970 catalog_manager.cc:4841] TS b3058bac42ec4f7286289fde5e1d43ce (127.3.204.65:38679): tablet 0cfdca07d5254d2c8a125a065a476849 (table test-table [id=04eaca9f612449e792388f2b8520926e]) successfully deleted
I20260811 12:33:20.831947 7022 ts_tablet_manager.cc:1918] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:20.832304 7022 log.cc:1196] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/0cfdca07d5254d2c8a125a065a476849
I20260811 12:33:20.832897 7022 ts_tablet_manager.cc:1939] T 0cfdca07d5254d2c8a125a065a476849 P 5afe16cda3a84d59b8c0c2d6b553ac13: Deleting consensus metadata
I20260811 12:33:20.834251 5971 catalog_manager.cc:4841] TS 5afe16cda3a84d59b8c0c2d6b553ac13 (127.3.204.66:33493): tablet 0cfdca07d5254d2c8a125a065a476849 (table test-table [id=04eaca9f612449e792388f2b8520926e]) successfully deleted
Aug 11 12:33:20 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:20.844231 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.795742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.3.204.65:38679)
0811 12:33:20.795954 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.795960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.822415 (+ 26455us) server_negotiation.cc:407] Connection header received
0811 12:33:20.822709 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.822716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.822817 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.822917 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:20.825728 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.827036 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.829080 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.829826 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.829931 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.836022 (+ 6091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:20.836047 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:20.836119 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:20.836194 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:20.840662 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.841700 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.841711 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.841715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.841971 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.842428 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.842433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.842436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.842957 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:20.843288 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.843568 (+ 280us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.843802 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260811 12:33:20.845479 6354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35570
I20260811 12:33:20.868340 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.849628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.3.204.126:33047)
0811 12:33:20.849904 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.849911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.850363 (+ 452us) server_negotiation.cc:407] Connection header received
0811 12:33:20.850618 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.850624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.850707 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.850787 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:20.853140 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.854294 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.856396 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.857140 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.857283 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.860746 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:20.860756 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:20.860761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:20.860830 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:20.864848 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.865788 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.865794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.865798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.865962 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.866421 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.866426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.866430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.866831 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:20.867093 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.867484 (+ 391us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.867736 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260811 12:33:20.868980 6000 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44892
I20260811 12:33:20.871402 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6115
I20260811 12:33:20.905766 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.874042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.3.204.65:38679)
0811 12:33:20.874261 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.874279 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.883089 (+ 8810us) server_negotiation.cc:407] Connection header received
0811 12:33:20.883297 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.883303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.883393 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.883506 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:20.890122 (+ 6616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.891261 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.893238 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.893910 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.894032 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.897462 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:20.897473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:20.897479 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:20.897526 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:20.901643 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.902774 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.902781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.902785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.902987 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.903680 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.903686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.903690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.904154 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:20.904537 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.905151 (+ 614us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.905408 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 12:33:20.907218 6354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35580
W20260811 12:33:20.907887 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6115: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:20.908145 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 6115
I20260811 12:33:20.945108 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6367
Aug 11 12:33:20 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:33:20.962349 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.945868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.3.204.66:33493)
0811 12:33:20.946060 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.946065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.946098 (+ 33us) server_negotiation.cc:407] Connection header received
0811 12:33:20.946293 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.946297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.946374 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.946462 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:20.948026 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.948992 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.950859 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.951268 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.951363 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:20.956413 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:20.956431 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:20.956448 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:20.956519 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:20.959446 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.960274 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.960285 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.960289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.960474 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:20.960895 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:20.960898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:20.960900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:20.961347 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:20.961608 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:20.961905 (+ 297us) server_negotiation.cc:299] Negotiation successful
0811 12:33:20.962099 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260811 12:33:20.963025 6608 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46484
W20260811 12:33:20.963863 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6367: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:20.964061 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 6367
I20260811 12:33:20.993039 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6622
Aug 11 12:33:20 dist-test-slave-lgr5 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:33:21.008155 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:20.993788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.3.204.67:45555)
0811 12:33:20.994220 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:20.994224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:20.994242 (+ 18us) server_negotiation.cc:407] Connection header received
0811 12:33:20.994305 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:20.994309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:20.994370 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:20.994466 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:20.995800 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.996556 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.997847 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:20.998258 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:20.998353 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:21.002507 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:21.002526 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:21.002544 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:21.002588 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:21.005619 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:21.006327 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:21.006333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:21.006336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:21.006472 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:21.006757 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:21.006760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:21.006762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:21.007147 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:21.007371 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:21.007653 (+ 282us) server_negotiation.cc:299] Negotiation successful
0811 12:33:21.007837 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
W20260811 12:33:21.008860 6839 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36358
W20260811 12:33:21.009488 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6622: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:21.009634 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 6622
I20260811 12:33:21.036057 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 5938
I20260811 12:33:21.037390 6065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:21.154382 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 5938
2026-08-11T12:33:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5368 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:21 dist-test-slave-lgr5 krb5kdc[7033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:21 dist-test-slave-lgr5 krb5kdc[7033](info): set up 2 sockets
Aug 11 12:33:21 dist-test-slave-lgr5 krb5kdc[7033](info): commencing operation
krb5kdc: starting...
W20260811 12:33:22.396965 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:22 dist-test-slave-lgr5 krb5kdc[7033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:22.603268 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:35433
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:35433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:22.934376 7055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:22.934681 7055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:22.934811 7055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:22.942196 7055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:22.942319 7055 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 12:33:22.942390 7055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:22.942483 7055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:22.942549 7055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:22.942637 7055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:22.952901 7055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:35433
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:35433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:22.954205 7055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:22.955570 7055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:22.967650 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:22.967715 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:22.967684 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:22.968534 7055 server_base.cc:1034] running on GCE node
I20260811 12:33:22.969311 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:22.970736 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:22.971968 7055 hybrid_clock.cc:648] HybridClock initialized: now 1786451602971892 us; error 44 us; skew 500 ppm
I20260811 12:33:22.972344 7055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:22 dist-test-slave-lgr5 krb5kdc[7033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:22.975565 7055 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:22.977216 7055 webserver.cc:460] Webserver started at http://127.3.204.126:43171/ using document root <none> and password file <none>
I20260811 12:33:22.977967 7055 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:22.978076 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:22.978394 7055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:22.981467 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "f83fef6e626344cead90218557fb9ad1"
format_stamp: "Formatted at 2026-08-11 12:33:22 on dist-test-slave-lgr5"
server_key: "3f9fa029c7dccdd0736165fcff5644ff"
server_key_iv: "d4fda2f567ae586f864d27f51f5f2274"
server_key_version: "encryptionkey@0"
I20260811 12:33:22.982276 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "f83fef6e626344cead90218557fb9ad1"
format_stamp: "Formatted at 2026-08-11 12:33:22 on dist-test-slave-lgr5"
server_key: "3f9fa029c7dccdd0736165fcff5644ff"
server_key_iv: "d4fda2f567ae586f864d27f51f5f2274"
server_key_version: "encryptionkey@0"
I20260811 12:33:22.987409 7055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260811 12:33:22.991037 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:22.992470 7055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:33:22.992646 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "f83fef6e626344cead90218557fb9ad1"
format_stamp: "Formatted at 2026-08-11 12:33:22 on dist-test-slave-lgr5"
server_key: "3f9fa029c7dccdd0736165fcff5644ff"
server_key_iv: "d4fda2f567ae586f864d27f51f5f2274"
server_key_version: "encryptionkey@0"
I20260811 12:33:22.992870 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:23.020805 7055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:23.024974 7055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:23.025246 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:23.046052 7055 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:35433
I20260811 12:33:23.046059 7188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:35433 every 8 connection(s)
I20260811 12:33:23.047482 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:23.051499 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:23.053705 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7055
I20260811 12:33:23.053946 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:23.054629 3889 external_mini_cluster.cc:1428] Setting key 15b58a03edf6e7fa594b4fd6d57c6ed5
I20260811 12:33:23.064316 7189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: Bootstrap starting.
I20260811 12:33:23.067658 7189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:23.068816 7189 log.cc:824] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: Log is configured to *not* fsync() on all Append() calls
Aug 11 12:33:23 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:23.071858 7189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: No bootstrap required, opened a new log
I20260811 12:33:23.077831 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:23.056926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.3.204.126:35433)
0811 12:33:23.057914 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:23.057930 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:23.057979 (+ 49us) server_negotiation.cc:407] Connection header received
0811 12:33:23.059756 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:23.059782 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:23.060268 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:23.060706 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:23.062611 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.063989 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.065358 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.066096 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.066278 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:23.070447 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:23.070456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:23.070474 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:23.070512 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:23.073574 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:23.074300 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:23.074307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:23.074315 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:23.074503 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:23.074905 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:23.074909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:23.074911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:23.075505 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:23.075792 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:23.076326 (+ 534us) server_negotiation.cc:299] Negotiation successful
0811 12:33:23.076765 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":455,"threads_started":1}
I20260811 12:33:23.080065 7189 raft_consensus.cc:348] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83fef6e626344cead90218557fb9ad1" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 35433 } }
I20260811 12:33:23.080567 7189 raft_consensus.cc:374] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:23.080680 7189 raft_consensus.cc:729] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83fef6e626344cead90218557fb9ad1, State: Initialized, Role: FOLLOWER
I20260811 12:33:23.081244 7189 consensus_queue.cc:260] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83fef6e626344cead90218557fb9ad1" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 35433 } }
I20260811 12:33:23.081480 7189 raft_consensus.cc:388] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:23.081611 7189 raft_consensus.cc:482] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:23.081779 7189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:23.083985 7189 raft_consensus.cc:504] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83fef6e626344cead90218557fb9ad1" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 35433 } }
I20260811 12:33:23.084546 7189 leader_election.cc:302] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f83fef6e626344cead90218557fb9ad1; no voters:
I20260811 12:33:23.085143 7189 leader_election.cc:288] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:23.085309 7196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:23.086668 7196 raft_consensus.cc:686] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [term 1 LEADER]: Becoming Leader. State: Replica: f83fef6e626344cead90218557fb9ad1, State: Running, Role: LEADER
I20260811 12:33:23.087239 7196 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83fef6e626344cead90218557fb9ad1" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 35433 } }
I20260811 12:33:23.087455 7189 sys_catalog.cc:564] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:23.090229 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f83fef6e626344cead90218557fb9ad1. Latest consensus state: current_term: 1 leader_uuid: "f83fef6e626344cead90218557fb9ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83fef6e626344cead90218557fb9ad1" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 35433 } } }
I20260811 12:33:23.090497 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f83fef6e626344cead90218557fb9ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83fef6e626344cead90218557fb9ad1" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 35433 } } }
I20260811 12:33:23.090736 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:23.090821 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:23.092792 7209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:23.099431 7209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:23.108192 7209 catalog_manager.cc:1347] Generated new cluster ID: 02a6fad4fe1e4caba15e7cb2bb01408f
I20260811 12:33:23.108333 7209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:23.126552 7209 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:23.127946 7209 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:23.138619 7209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: Generated new TSK 0
I20260811 12:33:23.139672 7209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:23.205838 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:35433
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:23.561645 7228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:23.561965 7228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:23.562078 7228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:23.569496 7228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:23.569622 7228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:23.569741 7228 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 12:33:23.569809 7228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:33:23.569895 7228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:23.580201 7228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:35433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:23.581566 7228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:23.582930 7228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:23.596370 7239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:23.596535 7238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:23.596392 7242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:23.601765 7228 server_base.cc:1034] running on GCE node
I20260811 12:33:23.602537 7228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:23.603608 7228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:23.604859 7228 hybrid_clock.cc:648] HybridClock initialized: now 1786451603604776 us; error 58 us; skew 500 ppm
I20260811 12:33:23.605211 7228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:23 dist-test-slave-lgr5 krb5kdc[7033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:23.611693 7228 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:23.613363 7228 webserver.cc:460] Webserver started at http://127.3.204.65:36791/ using document root <none> and password file <none>
I20260811 12:33:23.614105 7228 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:23.614230 7228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:23.614554 7228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:23.617607 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "02b2237deace4028b90bea06f6b4af67"
format_stamp: "Formatted at 2026-08-11 12:33:23 on dist-test-slave-lgr5"
server_key: "127d387f4f8b28a6965b5081399522e0"
server_key_iv: "84ce946895c741b7940bdee23eeb7aec"
server_key_version: "encryptionkey@0"
I20260811 12:33:23.618360 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "02b2237deace4028b90bea06f6b4af67"
format_stamp: "Formatted at 2026-08-11 12:33:23 on dist-test-slave-lgr5"
server_key: "127d387f4f8b28a6965b5081399522e0"
server_key_iv: "84ce946895c741b7940bdee23eeb7aec"
server_key_version: "encryptionkey@0"
I20260811 12:33:23.623797 7228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:33:23.627458 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:23.628896 7228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:23.629069 7228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "02b2237deace4028b90bea06f6b4af67"
format_stamp: "Formatted at 2026-08-11 12:33:23 on dist-test-slave-lgr5"
server_key: "127d387f4f8b28a6965b5081399522e0"
server_key_iv: "84ce946895c741b7940bdee23eeb7aec"
server_key_version: "encryptionkey@0"
I20260811 12:33:23.629262 7228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:23.654796 7228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:23.658988 7228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:23.659266 7228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:23.662232 7228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:23.662370 7228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:23.662508 7228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:23.663159 7228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:33:23 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:23.686244 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:23.662782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:37685 (local address 127.3.204.126:35433)
0811 12:33:23.663232 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:23.663238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:23.665119 (+ 1881us) server_negotiation.cc:407] Connection header received
0811 12:33:23.667185 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:23.667191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:23.667279 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:23.667394 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:23.669990 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.670927 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.673435 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.673874 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.674024 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:23.679313 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:23.679322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:23.679328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:23.679382 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:23.682653 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:23.683772 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:23.683777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:23.683779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:23.683909 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:23.684416 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:23.684420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:23.684423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:23.684717 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:23.684856 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:23.685633 (+ 777us) server_negotiation.cc:299] Negotiation successful
0811 12:33:23.685868 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":216,"threads_started":1}
I20260811 12:33:23.687273 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:23.663937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:35433 (local address 127.3.204.65:37685)
0811 12:33:23.664911 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:23.664964 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:23.666885 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:23.667720 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:23.667734 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:23.668700 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:23.669733 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:23.669767 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.671128 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.671132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:23.673235 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:23.673257 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.674198 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.674207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:23.674595 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:23.675471 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:23.675509 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:23.678983 (+ 3474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:23.682863 (+ 3880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:23.682878 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:23.682902 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:23.683627 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:23.684019 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:23.684022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:23.684025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:23.684282 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:23.684846 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:23.684854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:23.685240 (+ 386us) client_negotiation.cc:769] Sending connection context
0811 12:33:23.685502 (+ 262us) client_negotiation.cc:240] Negotiation successful
0811 12:33:23.686059 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":421,"threads_started":1}
I20260811 12:33:23.708622 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:23.693451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:49819 (local address 127.3.204.126:35433)
0811 12:33:23.693630 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:23.693636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:23.693921 (+ 285us) server_negotiation.cc:407] Connection header received
0811 12:33:23.694112 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:23.694116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:23.694178 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:23.694257 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:23.695527 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.696293 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.698655 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.699324 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.699473 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:23.702156 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:23.702165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:23.702170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:23.702215 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:23.705114 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:23.706207 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:23.706213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:23.706217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:23.706449 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:23.706917 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:23.706922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:23.706925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:23.707240 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:23.707379 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:23.707965 (+ 586us) server_negotiation.cc:299] Negotiation successful
0811 12:33:23.708257 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 12:33:23.708868 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:23.693185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:35433 (local address 127.3.204.65:49819)
0811 12:33:23.693810 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:23.693833 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:23.694005 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:23.694302 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:23.694307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:23.694986 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:23.695392 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:23.695406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.696505 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.696512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:23.698483 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:23.698501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:23.699568 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:23.699576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:23.699786 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:23.700581 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:23.700607 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:23.701929 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:23.705276 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:23.705283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:23.705288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:23.706064 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:23.706593 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:23.706598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:23.706602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:23.706789 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:23.707365 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:23.707371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:23.707632 (+ 261us) client_negotiation.cc:769] Sending connection context
0811 12:33:23.707822 (+ 190us) client_negotiation.cc:240] Negotiation successful
0811 12:33:23.708300 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":326,"threads_started":1}
I20260811 12:33:23.713222 7228 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:34373
I20260811 12:33:23.713263 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:34373 every 8 connection(s)
I20260811 12:33:23.714471 7228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:23.716655 7471 heartbeater.cc:347] Connected to a master server at 127.3.204.126:35433
I20260811 12:33:23.716864 7471 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:23.717473 7471 heartbeater.cc:511] Master 127.3.204.126:35433 requested a full tablet report, sending...
I20260811 12:33:23.719645 7112 ts_manager.cc:194] Registered new tserver with Master: 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373)
I20260811 12:33:23.720602 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7228
I20260811 12:33:23.720781 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:23.721300 7112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:37685
I20260811 12:33:23.721403 3889 external_mini_cluster.cc:1428] Setting key 3857125565a1028cbc717aab13bf08ca
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:23.767272 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:35433
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:24.101684 7476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:24.102023 7476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:24.102181 7476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:24.109652 7476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:24.109787 7476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:24.109922 7476 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 12:33:24.109990 7476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:33:24.110098 7476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:24.120302 7476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:35433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:24.121711 7476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:24.123041 7476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:24.136320 7491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:24.136365 7488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:24.136416 7496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:24.137251 7476 server_base.cc:1034] running on GCE node
I20260811 12:33:24.137903 7476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:24.138937 7476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:24.140206 7476 hybrid_clock.cc:648] HybridClock initialized: now 1786451604140136 us; error 54 us; skew 500 ppm
I20260811 12:33:24.140544 7476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:24 dist-test-slave-lgr5 krb5kdc[7033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:24.143756 7476 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:24.145303 7476 webserver.cc:460] Webserver started at http://127.3.204.66:39321/ using document root <none> and password file <none>
I20260811 12:33:24.146106 7476 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:24.146229 7476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:24.146544 7476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:24.149585 7476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "23b7060a4c9247b6b6d6122fef7b0092"
format_stamp: "Formatted at 2026-08-11 12:33:24 on dist-test-slave-lgr5"
server_key: "59bd25e8e36f3333de634d1a8f142648"
server_key_iv: "3efd163388ed5cbfbfc339ab258278d9"
server_key_version: "encryptionkey@0"
I20260811 12:33:24.150367 7476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "23b7060a4c9247b6b6d6122fef7b0092"
format_stamp: "Formatted at 2026-08-11 12:33:24 on dist-test-slave-lgr5"
server_key: "59bd25e8e36f3333de634d1a8f142648"
server_key_iv: "3efd163388ed5cbfbfc339ab258278d9"
server_key_version: "encryptionkey@0"
I20260811 12:33:24.155733 7476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:24.159610 7507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:24.161175 7476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:24.161362 7476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "23b7060a4c9247b6b6d6122fef7b0092"
format_stamp: "Formatted at 2026-08-11 12:33:24 on dist-test-slave-lgr5"
server_key: "59bd25e8e36f3333de634d1a8f142648"
server_key_iv: "3efd163388ed5cbfbfc339ab258278d9"
server_key_version: "encryptionkey@0"
I20260811 12:33:24.161576 7476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:24.181159 7476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:24.185328 7476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:24.185626 7476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:24.188633 7476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:24.188766 7476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:24.188895 7476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:24.188982 7476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:33:24 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:24.213199 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.189316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:60553 (local address 127.3.204.126:35433)
0811 12:33:24.189585 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.189592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.191141 (+ 1549us) server_negotiation.cc:407] Connection header received
0811 12:33:24.193448 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.193456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.193548 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.193657 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:24.195868 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.197103 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.199508 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.200121 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.200252 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.205587 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:24.205594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:24.205600 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:24.205646 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:24.208844 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.210067 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.210074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.210078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.210211 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.210832 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.210838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.210841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.211217 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:24.211362 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.212503 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.212779 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260811 12:33:24.214242 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.190085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:35433 (local address 127.3.204.66:60553)
0811 12:33:24.190980 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.191020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.193142 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.193972 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.193984 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.194765 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:24.195629 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.195668 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.197283 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.197289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.199356 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.199369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.200371 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.200382 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.200673 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.201728 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:24.201759 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:24.205259 (+ 3500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:24.209040 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:24.209056 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:24.209080 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:24.209910 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:24.210343 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:24.210348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:24.210352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:24.210706 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:24.211345 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:24.211357 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:24.211927 (+ 570us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.212367 (+ 440us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.212948 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":351,"threads_started":1}
I20260811 12:33:24.234853 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.220468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:45347 (local address 127.3.204.126:35433)
0811 12:33:24.220628 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.220632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.221213 (+ 581us) server_negotiation.cc:407] Connection header received
0811 12:33:24.221350 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.221356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.221462 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.221573 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:24.222466 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.223642 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.225516 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.226165 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.226307 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.228552 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:24.228560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:24.228565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:24.228615 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:24.231899 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.232929 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.232936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.232939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.233081 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.233504 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.233508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.233510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.233875 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:24.233988 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.234450 (+ 462us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.234618 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 12:33:24.235270 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.220532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:35433 (local address 127.3.204.66:45347)
0811 12:33:24.221105 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.221121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.221243 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.221588 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.221591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.222027 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:24.222316 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.222325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.223810 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.223814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.225263 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.225277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.226347 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.226353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.226465 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.227106 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:24.227130 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:24.228361 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:24.232045 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:24.232051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:24.232056 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:24.232778 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:24.233186 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:24.233191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:24.233195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:24.233391 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:24.233972 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:24.233977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:24.234206 (+ 229us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.234376 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.234740 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":304,"threads_started":1}
I20260811 12:33:24.240026 7476 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:46579
I20260811 12:33:24.240119 7719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:46579 every 8 connection(s)
I20260811 12:33:24.241262 7476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:24.243268 7721 heartbeater.cc:347] Connected to a master server at 127.3.204.126:35433
I20260811 12:33:24.243491 7721 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:24.244191 7721 heartbeater.cc:511] Master 127.3.204.126:35433 requested a full tablet report, sending...
I20260811 12:33:24.245576 7112 ts_manager.cc:194] Registered new tserver with Master: 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579)
I20260811 12:33:24.246411 7112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:60553
I20260811 12:33:24.249153 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7476
I20260811 12:33:24.249294 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:24.249687 3889 external_mini_cluster.cc:1428] Setting key 73970fc2c9451919f4496730a53e0c62
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:24.295280 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:35433
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:24.626032 7727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:24.626315 7727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:24.626446 7727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:24.633827 7727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:24.633940 7727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:24.634068 7727 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 12:33:24.634127 7727 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:33:24.634232 7727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:24.644237 7727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:35433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:24.645571 7727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:24.646879 7727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:24.660521 7735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:24.660521 7737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:24.660606 7740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:24.661571 7727 server_base.cc:1034] running on GCE node
I20260811 12:33:24.662261 7727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:24.663232 7727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:24.664439 7727 hybrid_clock.cc:648] HybridClock initialized: now 1786451604664387 us; error 36 us; skew 500 ppm
I20260811 12:33:24.664779 7727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:24 dist-test-slave-lgr5 krb5kdc[7033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:24.668045 7727 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:24.669704 7727 webserver.cc:460] Webserver started at http://127.3.204.67:32997/ using document root <none> and password file <none>
I20260811 12:33:24.670414 7727 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:24.670552 7727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:24.670881 7727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:24.673990 7727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "63ad2cd4cdc043f4a2b131e636497d2d"
format_stamp: "Formatted at 2026-08-11 12:33:24 on dist-test-slave-lgr5"
server_key: "34925cb2dbd32f8bb8f73ad003b03298"
server_key_iv: "13af5accdd37cf86cc8c04f4b9222af5"
server_key_version: "encryptionkey@0"
I20260811 12:33:24.674754 7727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "63ad2cd4cdc043f4a2b131e636497d2d"
format_stamp: "Formatted at 2026-08-11 12:33:24 on dist-test-slave-lgr5"
server_key: "34925cb2dbd32f8bb8f73ad003b03298"
server_key_iv: "13af5accdd37cf86cc8c04f4b9222af5"
server_key_version: "encryptionkey@0"
I20260811 12:33:24.680416 7727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260811 12:33:24.683979 7751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:24.685428 7727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:33:24.685593 7727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "63ad2cd4cdc043f4a2b131e636497d2d"
format_stamp: "Formatted at 2026-08-11 12:33:24 on dist-test-slave-lgr5"
server_key: "34925cb2dbd32f8bb8f73ad003b03298"
server_key_iv: "13af5accdd37cf86cc8c04f4b9222af5"
server_key_version: "encryptionkey@0"
I20260811 12:33:24.685787 7727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:24.713532 7727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:24.718117 7727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:24.718394 7727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:24.721160 7727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:24.721282 7727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:24.721474 7727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:24.721570 7727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:24.724325 7471 heartbeater.cc:503] Master 127.3.204.126:35433 was elected leader, sending a full tablet report...
Aug 11 12:33:24 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:24.746018 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.721598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:36943 (local address 127.3.204.126:35433)
0811 12:33:24.721789 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.721796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.723324 (+ 1528us) server_negotiation.cc:407] Connection header received
0811 12:33:24.725689 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.725695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.725782 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.725914 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:24.728184 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.729262 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.731968 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.732648 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.732782 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.737891 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:24.737899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:24.737904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:24.737949 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:24.741789 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.742925 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.742930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.742933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.743119 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.743793 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.743799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.743802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.744195 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:24.744343 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.745381 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.745642 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:33:24.747961 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.722341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:35433 (local address 127.3.204.67:36943)
0811 12:33:24.723155 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.723192 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.725291 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.726184 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.726195 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.726962 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:24.727927 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.727953 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.729520 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.729523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.731803 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.731823 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.732934 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.732945 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.733298 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.734160 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:24.734196 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:24.737628 (+ 3432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:24.741981 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:24.741996 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:24.742019 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:24.742777 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:24.743262 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:24.743268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:24.743271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:24.743637 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:24.744337 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:24.744346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:24.744881 (+ 535us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.745235 (+ 354us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.745822 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":356,"threads_started":1}
I20260811 12:33:24.769673 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.754470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:35433 (local address 127.3.204.67:34767)
0811 12:33:24.755169 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.755192 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.755356 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.755666 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.755671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.756346 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:24.756780 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.756796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.757885 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.757891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.759604 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.759621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.760482 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.760493 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.760672 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.761475 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:24.761503 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:24.762896 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:24.766613 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:24.766619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:24.766622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:24.767197 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:24.767644 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:24.767650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:24.767655 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:24.767811 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:24.768449 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:24.768454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:24.768749 (+ 295us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.768961 (+ 212us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.769325 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":343,"threads_started":1}
I20260811 12:33:24.769779 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.754437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:34767 (local address 127.3.204.126:35433)
0811 12:33:24.754636 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.754643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.755292 (+ 649us) server_negotiation.cc:407] Connection header received
0811 12:33:24.755472 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.755476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.755539 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.755618 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:24.756974 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.757754 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.759782 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.760266 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.760398 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.763088 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:24.763093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:24.763096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:24.763130 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:24.766443 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.767339 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.767346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.767349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.767520 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.767931 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.767936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.767939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.768314 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:24.768470 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.769092 (+ 622us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.769384 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:33:24.777276 7727 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:40973
I20260811 12:33:24.777316 7950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:40973 every 8 connection(s)
I20260811 12:33:24.778527 7727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:24.780587 7952 heartbeater.cc:347] Connected to a master server at 127.3.204.126:35433
I20260811 12:33:24.780808 7952 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:24.781440 7952 heartbeater.cc:511] Master 127.3.204.126:35433 requested a full tablet report, sending...
I20260811 12:33:24.782878 7112 ts_manager.cc:194] Registered new tserver with Master: 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:24.783717 7112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:36943
I20260811 12:33:24.787287 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7727
I20260811 12:33:24.787436 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:24.787853 3889 external_mini_cluster.cc:1428] Setting key 1eb87698f1f905a192dd10fa299a18b2
I20260811 12:33:24.790824 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 12:33:24 dist-test-slave-lgr5 krb5kdc[7033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 12:33:24 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:24.841588 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.826860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.3.204.126:35433)
0811 12:33:24.827091 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.827098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.827275 (+ 177us) server_negotiation.cc:407] Connection header received
0811 12:33:24.827511 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.827516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.827583 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.827673 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:24.829312 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.830052 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.831324 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.831765 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.831894 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.835982 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:24.835988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:24.835992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:24.836035 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:24.838676 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.839399 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.839404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.839406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.839550 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:24.839948 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:24.839959 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:24.839961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:24.840335 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:24.840592 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.840905 (+ 313us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.841222 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 12:33:24.846091 7112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:24.853201 7112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:24.881680 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.866391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46579 (local address 127.0.0.1:45526)
0811 12:33:24.867896 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.867944 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.868184 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.870673 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.870681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.870717 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:24.871178 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.871192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.874788 (+ 3596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.874795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.878715 (+ 3920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.878732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.880658 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.880664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.880891 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.880958 (+ 67us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.881080 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.881289 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"threads_started":1}
I20260811 12:33:24.883026 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.866567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.3.204.66:46579)
0811 12:33:24.868754 (+ 2187us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.868763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.868791 (+ 28us) server_negotiation.cc:407] Connection header received
0811 12:33:24.868889 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.868897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.869178 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.871429 (+ 2251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:24.871675 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.874619 (+ 2944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.878954 (+ 4335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.880419 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.881956 (+ 1537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.882208 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.882288 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.882733 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2036,"thread_start_us":1906,"threads_started":1}
I20260811 12:33:24.887934 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.866697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34373 (local address 127.0.0.1:48392)
0811 12:33:24.878104 (+ 11407us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.878126 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.878247 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.879798 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.879803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.879830 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:24.880267 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.880281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.882113 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.882119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.885024 (+ 2905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.885041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.887153 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.887160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.887333 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.887366 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.887449 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.887595 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11241,"thread_start_us":768,"threads_started":1}
I20260811 12:33:24.887099 7601 tablet_service.cc:1467] Processing CreateTablet for tablet 99eae7e5f3bc464397e838605348629e (DEFAULT_TABLE table=test-table [id=e2f9846c2baf4aa297216a128cf4b115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:24.889050 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.867321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.3.204.65:34373)
0811 12:33:24.878909 (+ 11588us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.878919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.878950 (+ 31us) server_negotiation.cc:407] Connection header received
0811 12:33:24.879073 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.879081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.879361 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.879581 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:24.880465 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.881907 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.885217 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.886638 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.886775 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.886996 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.888376 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.888644 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11408,"thread_start_us":11270,"threads_started":1}
I20260811 12:33:24.890139 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99eae7e5f3bc464397e838605348629e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:24.890302 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.869394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:40973 (local address 127.0.0.1:59332)
0811 12:33:24.881282 (+ 11888us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.881305 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.881449 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.882983 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.882987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.883018 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:24.883360 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.883372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.885340 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.885345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.887011 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.887028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.889050 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.889058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.889519 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.889555 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.889646 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.889938 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11742,"thread_start_us":8934,"threads_started":1}
I20260811 12:33:24.890879 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.869934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.3.204.67:40973)
0811 12:33:24.882300 (+ 12366us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.882310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.882347 (+ 37us) server_negotiation.cc:407] Connection header received
0811 12:33:24.882457 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.882466 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.882761 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.883583 (+ 822us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:24.883864 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.885179 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.887277 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.888789 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.888937 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.889187 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.890413 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.890603 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12184,"thread_start_us":12088,"threads_started":1}
I20260811 12:33:24.893615 7336 tablet_service.cc:1467] Processing CreateTablet for tablet 99eae7e5f3bc464397e838605348629e (DEFAULT_TABLE table=test-table [id=e2f9846c2baf4aa297216a128cf4b115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:24.894637 7829 tablet_service.cc:1467] Processing CreateTablet for tablet 99eae7e5f3bc464397e838605348629e (DEFAULT_TABLE table=test-table [id=e2f9846c2baf4aa297216a128cf4b115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:24.895257 7336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99eae7e5f3bc464397e838605348629e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:24.896117 7829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99eae7e5f3bc464397e838605348629e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:24.904786 7980 tablet_bootstrap.cc:492] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Bootstrap starting.
I20260811 12:33:24.908915 7980 tablet_bootstrap.cc:654] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:24.909943 7980 log.cc:824] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:24.910876 7983 tablet_bootstrap.cc:492] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Bootstrap starting.
I20260811 12:33:24.912161 7981 tablet_bootstrap.cc:492] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Bootstrap starting.
I20260811 12:33:24.914654 7983 tablet_bootstrap.cc:654] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:24.915640 7983 log.cc:824] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:24.915840 7981 tablet_bootstrap.cc:654] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:24.918040 7981 log.cc:824] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:24.918686 7980 tablet_bootstrap.cc:492] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: No bootstrap required, opened a new log
I20260811 12:33:24.918682 7983 tablet_bootstrap.cc:492] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: No bootstrap required, opened a new log
I20260811 12:33:24.918972 7983 ts_tablet_manager.cc:1397] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260811 12:33:24.918972 7980 ts_tablet_manager.cc:1397] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260811 12:33:24.920480 7981 tablet_bootstrap.cc:492] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: No bootstrap required, opened a new log
I20260811 12:33:24.920711 7981 ts_tablet_manager.cc:1397] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260811 12:33:24.927627 7981 raft_consensus.cc:348] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:24.928018 7981 raft_consensus.cc:374] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:24.928139 7981 raft_consensus.cc:729] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b2237deace4028b90bea06f6b4af67, State: Initialized, Role: FOLLOWER
I20260811 12:33:24.928764 7981 consensus_queue.cc:260] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:24.929445 7980 raft_consensus.cc:348] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:24.929440 7983 raft_consensus.cc:348] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:24.929867 7983 raft_consensus.cc:374] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:24.929867 7980 raft_consensus.cc:374] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:24.929961 7983 raft_consensus.cc:729] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ad2cd4cdc043f4a2b131e636497d2d, State: Initialized, Role: FOLLOWER
I20260811 12:33:24.929961 7980 raft_consensus.cc:729] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b7060a4c9247b6b6d6122fef7b0092, State: Initialized, Role: FOLLOWER
I20260811 12:33:24.930641 7980 consensus_queue.cc:260] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:24.930641 7983 consensus_queue.cc:260] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:24.932351 7981 ts_tablet_manager.cc:1428] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260811 12:33:24.933517 7952 heartbeater.cc:503] Master 127.3.204.126:35433 was elected leader, sending a full tablet report...
I20260811 12:33:24.934199 7983 ts_tablet_manager.cc:1428] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260811 12:33:24.935840 7721 heartbeater.cc:503] Master 127.3.204.126:35433 was elected leader, sending a full tablet report...
I20260811 12:33:24.936662 7980 ts_tablet_manager.cc:1428] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Time spent starting tablet: real 0.018s user 0.016s sys 0.003s
W20260811 12:33:24.966771 7472 tablet.cc:2369] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:33:24.983964 7989 raft_consensus.cc:482] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:24.984344 7989 raft_consensus.cc:504] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:24.986092 7989 leader_election.cc:288] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579), 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373)
I20260811 12:33:24.992787 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.986411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46579 (local address 127.3.204.67:50603)
0811 12:33:24.986788 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.986820 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.987004 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.987333 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.987337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.987365 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:24.987565 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.987573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.988982 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.988988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.990939 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.990955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.992211 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.992215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.992373 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.992401 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.992476 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.992575 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":27}
I20260811 12:33:24.992921 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.986552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:50603 (local address 127.3.204.66:46579)
0811 12:33:24.986763 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.986770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.986921 (+ 151us) server_negotiation.cc:407] Connection header received
0811 12:33:24.987151 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.987155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.987217 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.987292 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:24.987736 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.988767 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.991094 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.992043 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.992229 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.992306 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.992520 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.992653 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 12:33:24.993535 7722 tablet.cc:2369] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:33:24.993914 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99eae7e5f3bc464397e838605348629e" candidate_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b7060a4c9247b6b6d6122fef7b0092" is_pre_election: true
I20260811 12:33:24.994436 7632 raft_consensus.cc:2454] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ad2cd4cdc043f4a2b131e636497d2d in term 0.
I20260811 12:33:24.995098 7754 leader_election.cc:302] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b7060a4c9247b6b6d6122fef7b0092, 63ad2cd4cdc043f4a2b131e636497d2d; no voters:
I20260811 12:33:24.995223 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.986643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:59199 (local address 127.3.204.65:34373)
0811 12:33:24.986832 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:24.986837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:24.987421 (+ 584us) server_negotiation.cc:407] Connection header received
0811 12:33:24.987588 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:24.987592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:24.987660 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:24.987755 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:24.988434 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.989367 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.991728 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.994310 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.994483 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.994562 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:24.994853 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 12:33:24.994998 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 12:33:24.995410 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:24.986501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34373 (local address 127.3.204.67:59199)
0811 12:33:24.987282 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:24.987296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:24.987451 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:24.987818 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:24.987821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:24.987846 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:24.988257 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.988267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.989522 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.989527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.991580 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:24.991592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:24.994520 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:24.994527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:24.994699 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:24.994724 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 12:33:24.994809 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 12:33:24.994951 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":18,"thread_start_us":469,"threads_started":1}
I20260811 12:33:24.995546 7989 raft_consensus.cc:2781] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:24.995699 7989 raft_consensus.cc:482] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:24.995806 7989 raft_consensus.cc:3037] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:24.996516 7388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99eae7e5f3bc464397e838605348629e" candidate_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b2237deace4028b90bea06f6b4af67" is_pre_election: true
I20260811 12:33:24.997104 7388 raft_consensus.cc:2454] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ad2cd4cdc043f4a2b131e636497d2d in term 0.
I20260811 12:33:24.998718 7989 raft_consensus.cc:504] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:24.999279 7989 leader_election.cc:288] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [CANDIDATE]: Term 1 election: Requested vote from peers 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579), 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373)
I20260811 12:33:24.999763 7388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99eae7e5f3bc464397e838605348629e" candidate_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b2237deace4028b90bea06f6b4af67"
I20260811 12:33:24.999758 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99eae7e5f3bc464397e838605348629e" candidate_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b7060a4c9247b6b6d6122fef7b0092"
I20260811 12:33:24.999936 7388 raft_consensus.cc:3037] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:24.999953 7632 raft_consensus.cc:3037] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:25.002410 7388 raft_consensus.cc:2454] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ad2cd4cdc043f4a2b131e636497d2d in term 1.
I20260811 12:33:25.002411 7632 raft_consensus.cc:2454] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ad2cd4cdc043f4a2b131e636497d2d in term 1.
I20260811 12:33:25.002903 7758 leader_election.cc:302] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67, 63ad2cd4cdc043f4a2b131e636497d2d; no voters:
I20260811 12:33:25.003271 7989 raft_consensus.cc:2781] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:25.003823 7989 raft_consensus.cc:686] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 LEADER]: Becoming Leader. State: Replica: 63ad2cd4cdc043f4a2b131e636497d2d, State: Running, Role: LEADER
I20260811 12:33:25.004493 7989 consensus_queue.cc:237] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.009389 7108 catalog_manager.cc:5495] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67). New cstate: current_term: 1 leader_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } health_report { overall_health: HEALTHY } } }
W20260811 12:33:25.030073 7955 tablet.cc:2369] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 12:33:25 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:25.070461 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.054832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:35433 (local address 127.0.0.1:51826)
0811 12:33:25.055511 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:25.055529 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:25.055667 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:25.056112 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:25.056118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:25.056885 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:25.057204 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.057215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.058762 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.058767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.060297 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.060314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.061125 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.061132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.061324 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.062133 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:25.062157 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:25.064734 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:25.067811 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:25.067826 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:25.067832 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:25.068369 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:25.068751 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:25.068755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:25.068758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:25.068915 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:25.069547 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:25.069554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:25.069722 (+ 168us) client_negotiation.cc:769] Sending connection context
0811 12:33:25.069859 (+ 137us) client_negotiation.cc:240] Negotiation successful
0811 12:33:25.070085 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":382,"threads_started":1}
I20260811 12:33:25.070534 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.054781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.3.204.126:35433)
0811 12:33:25.054984 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.055009 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.055625 (+ 616us) server_negotiation.cc:407] Connection header received
0811 12:33:25.055795 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.055801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.055881 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.055993 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:25.057401 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.058593 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.060446 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.060893 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.060996 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.064961 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:25.064967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:25.064971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:25.065027 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:25.067654 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:25.068514 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:25.068518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:25.068521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:25.068651 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:25.069055 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:25.069059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:25.069061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:25.069407 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:25.069528 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.070019 (+ 491us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.070219 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 12:33:25.075789 7108 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:51826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 12:33:25.076936 7108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:25.083846 7601 tablet_service.cc:1467] Processing CreateTablet for tablet 2f733064cd1842f495c7708f233de5c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d90fd18ea764910b2569d51f1eea773]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:33:25.083788 7336 tablet_service.cc:1467] Processing CreateTablet for tablet 2f733064cd1842f495c7708f233de5c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d90fd18ea764910b2569d51f1eea773]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:33:25.084074 7829 tablet_service.cc:1467] Processing CreateTablet for tablet 2f733064cd1842f495c7708f233de5c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d90fd18ea764910b2569d51f1eea773]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:33:25.084916 7336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f733064cd1842f495c7708f233de5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:25.084916 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f733064cd1842f495c7708f233de5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:25.084949 7829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f733064cd1842f495c7708f233de5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:25.090909 7983 tablet_bootstrap.cc:492] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: Bootstrap starting.
I20260811 12:33:25.092237 7981 tablet_bootstrap.cc:492] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: Bootstrap starting.
I20260811 12:33:25.092530 7983 tablet_bootstrap.cc:654] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:25.092727 7980 tablet_bootstrap.cc:492] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: Bootstrap starting.
I20260811 12:33:25.093835 7981 tablet_bootstrap.cc:654] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:25.094080 7980 tablet_bootstrap.cc:654] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:25.095831 7983 tablet_bootstrap.cc:492] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: No bootstrap required, opened a new log
I20260811 12:33:25.095990 7983 ts_tablet_manager.cc:1397] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 12:33:25.096129 7981 tablet_bootstrap.cc:492] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: No bootstrap required, opened a new log
I20260811 12:33:25.096278 7980 tablet_bootstrap.cc:492] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: No bootstrap required, opened a new log
I20260811 12:33:25.096278 7981 ts_tablet_manager.cc:1397] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260811 12:33:25.096462 7980 ts_tablet_manager.cc:1397] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260811 12:33:25.096876 7983 raft_consensus.cc:348] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.097007 7981 raft_consensus.cc:348] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.097147 7983 raft_consensus.cc:374] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:25.097221 7981 raft_consensus.cc:374] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:25.097230 7983 raft_consensus.cc:729] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ad2cd4cdc043f4a2b131e636497d2d, State: Initialized, Role: FOLLOWER
I20260811 12:33:25.097316 7981 raft_consensus.cc:729] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b2237deace4028b90bea06f6b4af67, State: Initialized, Role: FOLLOWER
I20260811 12:33:25.097465 7983 consensus_queue.cc:260] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.097527 7981 consensus_queue.cc:260] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.097132 7980 raft_consensus.cc:348] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.098032 7983 ts_tablet_manager.cc:1428] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 12:33:25.098104 7981 ts_tablet_manager.cc:1428] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260811 12:33:25.098011 7988 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.098382 7988 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:25.098351 7989 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.098609 7989 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:25.099141 7980 raft_consensus.cc:374] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:25.099256 7980 raft_consensus.cc:729] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b7060a4c9247b6b6d6122fef7b0092, State: Initialized, Role: FOLLOWER
I20260811 12:33:25.099514 7980 consensus_queue.cc:260] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.100144 7980 ts_tablet_manager.cc:1428] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260811 12:33:25.100055 7991 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.100456 7991 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:25.222239 7991 raft_consensus.cc:482] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:25.222491 7991 raft_consensus.cc:504] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.224223 7991 leader_election.cc:288] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373), 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:25.231312 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.224596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:53421 (local address 127.3.204.65:34373)
0811 12:33:25.224772 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.224778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.225265 (+ 487us) server_negotiation.cc:407] Connection header received
0811 12:33:25.225473 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.225479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.225581 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.225691 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:25.226400 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.227355 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.229438 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.230269 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.230398 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.230452 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.230873 (+ 421us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.231020 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 12:33:25.231325 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.225439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:56469 (local address 127.3.204.67:40973)
0811 12:33:25.225617 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.225623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.225999 (+ 376us) server_negotiation.cc:407] Connection header received
0811 12:33:25.226189 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.226195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.226275 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.226372 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:25.226941 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.227729 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.229580 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.230391 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.230560 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.230637 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.230954 (+ 317us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.231090 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 12:33:25.231397 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.224514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34373 (local address 127.3.204.66:53421)
0811 12:33:25.225150 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:25.225175 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:25.225324 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:25.225756 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:25.225760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:25.225790 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:25.226234 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.226248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.227486 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.227492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.229228 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.229242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.230496 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.230503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.230715 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.230743 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 12:33:25.230823 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 12:33:25.230932 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":19,"spinlock_wait_cycles":46976,"thread_start_us":311,"threads_started":1}
I20260811 12:33:25.231444 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.225272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:40973 (local address 127.3.204.66:56469)
0811 12:33:25.225878 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:25.225891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:25.226048 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:25.226400 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:25.226404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:25.226430 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:25.226802 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.226814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.227867 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.227870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.229429 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.229443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.230630 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.230635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.230790 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.230822 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 12:33:25.230911 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 12:33:25.231029 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":22,"spinlock_wait_cycles":7680,"thread_start_us":326,"threads_started":1}
I20260811 12:33:25.232333 7388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f733064cd1842f495c7708f233de5c2" candidate_uuid: "23b7060a4c9247b6b6d6122fef7b0092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b2237deace4028b90bea06f6b4af67" is_pre_election: true
I20260811 12:33:25.232304 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f733064cd1842f495c7708f233de5c2" candidate_uuid: "23b7060a4c9247b6b6d6122fef7b0092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" is_pre_election: true
I20260811 12:33:25.232594 7388 raft_consensus.cc:2454] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b7060a4c9247b6b6d6122fef7b0092 in term 0.
I20260811 12:33:25.232697 7857 raft_consensus.cc:2454] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b7060a4c9247b6b6d6122fef7b0092 in term 0.
I20260811 12:33:25.233180 7512 leader_election.cc:302] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67, 23b7060a4c9247b6b6d6122fef7b0092; no voters:
I20260811 12:33:25.233606 7991 raft_consensus.cc:2781] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:25.233741 7991 raft_consensus.cc:482] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:25.233847 7991 raft_consensus.cc:3037] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:25.236116 7991 raft_consensus.cc:504] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.236734 7991 leader_election.cc:288] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [CANDIDATE]: Term 1 election: Requested vote from peers 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373), 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:25.237195 7388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f733064cd1842f495c7708f233de5c2" candidate_uuid: "23b7060a4c9247b6b6d6122fef7b0092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b2237deace4028b90bea06f6b4af67"
I20260811 12:33:25.237303 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f733064cd1842f495c7708f233de5c2" candidate_uuid: "23b7060a4c9247b6b6d6122fef7b0092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ad2cd4cdc043f4a2b131e636497d2d"
I20260811 12:33:25.237406 7388 raft_consensus.cc:3037] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:25.237514 7857 raft_consensus.cc:3037] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:25.239607 7388 raft_consensus.cc:2454] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b7060a4c9247b6b6d6122fef7b0092 in term 1.
I20260811 12:33:25.239681 7857 raft_consensus.cc:2454] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b7060a4c9247b6b6d6122fef7b0092 in term 1.
I20260811 12:33:25.240073 7512 leader_election.cc:302] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67, 23b7060a4c9247b6b6d6122fef7b0092; no voters:
I20260811 12:33:25.240410 7991 raft_consensus.cc:2781] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:25.240922 7991 raft_consensus.cc:686] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 LEADER]: Becoming Leader. State: Replica: 23b7060a4c9247b6b6d6122fef7b0092, State: Running, Role: LEADER
I20260811 12:33:25.241705 7991 consensus_queue.cc:237] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.246371 8010 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: TxnStatusTablet state changed. Reason: New leader 23b7060a4c9247b6b6d6122fef7b0092. Latest consensus state: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.246774 8010 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:33:25.247294 7112 catalog_manager.cc:5495] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66). New cstate: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } health_report { overall_health: UNKNOWN } } }
I20260811 12:33:25.247545 8013 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 12:33:25.247824 8013 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 12:33:25.298548 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.292139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46579 (local address 127.0.0.1:45536)
0811 12:33:25.292402 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:25.292424 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:25.292560 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:25.293035 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:25.293038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:25.293047 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:25.293220 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.293227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.294650 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.294654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.295686 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.295700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.296762 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.296770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.296926 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.297001 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:25.298103 (+ 1102us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:25.298120 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 12:33:25.298177 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 12:33:25.298306 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 12:33:25.298640 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.292374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.3.204.66:46579)
0811 12:33:25.292574 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.292581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.292603 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:33:25.292739 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.292746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.292861 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.293040 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:25.293345 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.294533 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.295839 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.296594 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.296757 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.297201 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:25.298006 (+ 805us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:25.298163 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.298213 (+ 50us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.298305 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 12:33:25.306764 7857 raft_consensus.cc:1261] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Refusing update from remote peer 23b7060a4c9247b6b6d6122fef7b0092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:25.306952 7388 raft_consensus.cc:1261] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Refusing update from remote peer 23b7060a4c9247b6b6d6122fef7b0092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:25.307814 8010 consensus_queue.cc:1035] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:25.308259 7991 consensus_queue.cc:1035] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:25.315861 7989 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: TxnStatusTablet state changed. Reason: New leader 23b7060a4c9247b6b6d6122fef7b0092. Latest consensus state: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.316232 7989 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:25.319911 8017 mvcc.cc:205] Tried to move back new op lower bound from 7317305775326588928 to 7317305775074131968. Current Snapshot: MvccSnapshot[applied={T|T < 7317305775326588928}]
I20260811 12:33:25.320317 8019 mvcc.cc:205] Tried to move back new op lower bound from 7317305775326588928 to 7317305775074131968. Current Snapshot: MvccSnapshot[applied={T|T < 7317305775326588928}]
I20260811 12:33:25.322925 7991 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.323257 7991 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:33:25.323695 7989 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.324021 7989 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 63ad2cd4cdc043f4a2b131e636497d2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:25.334300 7988 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: TxnStatusTablet state changed. Reason: New leader 23b7060a4c9247b6b6d6122fef7b0092. Latest consensus state: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.334602 7988 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:25.336355 8010 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.336650 8010 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 23b7060a4c9247b6b6d6122fef7b0092: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:33:25.340270 7988 tablet_replica.cc:439] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } } }
I20260811 12:33:25.340523 7988 tablet_replica.cc:442] T 2f733064cd1842f495c7708f233de5c2 P 02b2237deace4028b90bea06f6b4af67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:33:25.344822 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.336902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.3.204.67:40973)
0811 12:33:25.337088 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.337094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.337113 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:33:25.337216 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.337221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.337335 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.337495 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:25.338294 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.339473 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.341976 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.342622 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.342745 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.343424 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:25.344181 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:25.344343 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.344404 (+ 61us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.344515 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 12:33:25.362061 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.355815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:60363 (local address 127.3.204.66:46579)
0811 12:33:25.356044 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.356050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.356392 (+ 342us) server_negotiation.cc:407] Connection header received
0811 12:33:25.356600 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.356607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.356675 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.356759 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:25.357302 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.358065 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.360156 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.360648 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.360752 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.361329 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:25.361454 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:25.361530 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.361722 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.361804 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 12:33:25.362203 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.355632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46579 (local address 127.3.204.67:60363)
0811 12:33:25.356292 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:25.356312 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:25.356475 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:25.356787 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:25.356793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:25.356805 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:25.357143 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.357156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.358186 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.358192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.359936 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.359954 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.360829 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.360836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.361020 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.361106 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:25.361552 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:25.361563 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 12:33:25.361649 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 12:33:25.361804 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":278,"threads_started":1}
I20260811 12:33:25.373900 7388 raft_consensus.cc:1261] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Refusing update from remote peer 63ad2cd4cdc043f4a2b131e636497d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:25.374310 7632 raft_consensus.cc:1261] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Refusing update from remote peer 63ad2cd4cdc043f4a2b131e636497d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:25.374709 7989 consensus_queue.cc:1035] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:25.375144 7993 consensus_queue.cc:1035] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:25.389554 8020 mvcc.cc:205] Tried to move back new op lower bound from 7317305775603675136 to 7317305774102618112. Current Snapshot: MvccSnapshot[applied={T|T < 7317305775603675136}]
I20260811 12:33:25.426800 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.418129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:40973 (local address 127.3.204.66:59907)
0811 12:33:25.420527 (+ 2398us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:25.420543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:25.420664 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:25.421015 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:25.421018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:25.421029 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:25.421366 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.421375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.422888 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.422895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.424566 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.424584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.425628 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.425635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.425793 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.425865 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:25.426299 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:25.426308 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 12:33:25.426373 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 12:33:25.426478 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2204,"thread_start_us":2078,"threads_started":1}
I20260811 12:33:25.426784 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.418112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:59907 (local address 127.3.204.67:40973)
0811 12:33:25.418444 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.418451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.420630 (+ 2179us) server_negotiation.cc:407] Connection header received
0811 12:33:25.420786 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.420792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.420888 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.420978 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:25.421508 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.422579 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.424768 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.425447 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.425594 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.426054 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:25.426187 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:25.426286 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.426422 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.426508 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260811 12:33:25.468829 7112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:25.469148 7112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:25.473192 7112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: Sending DeleteTablet for 3 replicas of tablet 99eae7e5f3bc464397e838605348629e
I20260811 12:33:25.474413 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet 99eae7e5f3bc464397e838605348629e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:25 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:45526
I20260811 12:33:25.474510 7336 tablet_service.cc:1514] Processing DeleteTablet for tablet 99eae7e5f3bc464397e838605348629e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:25 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:48392
I20260811 12:33:25.475033 7829 tablet_service.cc:1514] Processing DeleteTablet for tablet 99eae7e5f3bc464397e838605348629e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:25 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:59332
I20260811 12:33:25.481201 7108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:25.481923 7108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:25.489341 8058 tablet_replica.cc:330] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: stopping tablet replica
I20260811 12:33:25.490006 8058 raft_consensus.cc:2229] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:33:25.490749 8058 raft_consensus.cc:2258] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:25.491586 7335 tablet_service.cc:1467] Processing CreateTablet for tablet e174fbc1eff54166999dd34abb0c5cda (DEFAULT_TABLE table=test-table [id=e275e4b9885145e69e3515503ceaff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:25.492182 7335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e174fbc1eff54166999dd34abb0c5cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:25.492451 7600 tablet_service.cc:1467] Processing CreateTablet for tablet e174fbc1eff54166999dd34abb0c5cda (DEFAULT_TABLE table=test-table [id=e275e4b9885145e69e3515503ceaff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:25.492836 7829 tablet_service.cc:1467] Processing CreateTablet for tablet e174fbc1eff54166999dd34abb0c5cda (DEFAULT_TABLE table=test-table [id=e275e4b9885145e69e3515503ceaff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:25.493008 7600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e174fbc1eff54166999dd34abb0c5cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:25.493387 7829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e174fbc1eff54166999dd34abb0c5cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:25.494666 8055 tablet_replica.cc:330] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: stopping tablet replica
I20260811 12:33:25.495164 8055 raft_consensus.cc:2229] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:25.495574 8055 raft_consensus.cc:2258] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:25.499516 7980 tablet_bootstrap.cc:492] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: Bootstrap starting.
I20260811 12:33:25.501312 7980 tablet_bootstrap.cc:654] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:25.503914 7983 tablet_bootstrap.cc:492] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: Bootstrap starting.
I20260811 12:33:25.505708 7983 tablet_bootstrap.cc:654] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:25.509614 8056 tablet_replica.cc:330] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: stopping tablet replica
I20260811 12:33:25.509692 8055 ts_tablet_manager.cc:1905] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:25.509977 7981 tablet_bootstrap.cc:492] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: Bootstrap starting.
I20260811 12:33:25.510259 8056 raft_consensus.cc:2229] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:25.510766 8056 raft_consensus.cc:2258] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:25.511799 7981 tablet_bootstrap.cc:654] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:25.512727 8056 ts_tablet_manager.cc:1905] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:25.517122 8055 ts_tablet_manager.cc:1918] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:25.517289 8055 log.cc:1196] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/99eae7e5f3bc464397e838605348629e
I20260811 12:33:25.517825 8055 ts_tablet_manager.cc:1939] T 99eae7e5f3bc464397e838605348629e P 02b2237deace4028b90bea06f6b4af67: Deleting consensus metadata
I20260811 12:33:25.519678 7091 catalog_manager.cc:4841] TS 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373): tablet 99eae7e5f3bc464397e838605348629e (table test-table [id=e2f9846c2baf4aa297216a128cf4b115]) successfully deleted
I20260811 12:33:25.520431 8056 ts_tablet_manager.cc:1918] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:25.520668 8056 log.cc:1196] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/99eae7e5f3bc464397e838605348629e
I20260811 12:33:25.521288 8056 ts_tablet_manager.cc:1939] T 99eae7e5f3bc464397e838605348629e P 23b7060a4c9247b6b6d6122fef7b0092: Deleting consensus metadata
I20260811 12:33:25.522099 7983 tablet_bootstrap.cc:492] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: No bootstrap required, opened a new log
I20260811 12:33:25.522292 7983 ts_tablet_manager.cc:1397] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20260811 12:33:25.523123 7983 raft_consensus.cc:348] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.523243 7089 catalog_manager.cc:4841] TS 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579): tablet 99eae7e5f3bc464397e838605348629e (table test-table [id=e2f9846c2baf4aa297216a128cf4b115]) successfully deleted
I20260811 12:33:25.523398 7983 raft_consensus.cc:374] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:25.523465 7983 raft_consensus.cc:729] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ad2cd4cdc043f4a2b131e636497d2d, State: Initialized, Role: FOLLOWER
I20260811 12:33:25.523743 7983 consensus_queue.cc:260] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.524379 7983 ts_tablet_manager.cc:1428] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260811 12:33:25.529973 8058 ts_tablet_manager.cc:1905] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:25.530982 7981 tablet_bootstrap.cc:492] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: No bootstrap required, opened a new log
I20260811 12:33:25.531176 7981 ts_tablet_manager.cc:1397] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.008s
I20260811 12:33:25.534662 7981 raft_consensus.cc:348] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.534945 7981 raft_consensus.cc:374] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:25.535053 7981 raft_consensus.cc:729] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b2237deace4028b90bea06f6b4af67, State: Initialized, Role: FOLLOWER
I20260811 12:33:25.536242 7980 tablet_bootstrap.cc:492] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: No bootstrap required, opened a new log
I20260811 12:33:25.536444 7980 ts_tablet_manager.cc:1397] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.004s
I20260811 12:33:25.537282 7980 raft_consensus.cc:348] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.537545 7980 raft_consensus.cc:374] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:25.537633 7980 raft_consensus.cc:729] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b7060a4c9247b6b6d6122fef7b0092, State: Initialized, Role: FOLLOWER
I20260811 12:33:25.537940 7980 consensus_queue.cc:260] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.538506 8058 ts_tablet_manager.cc:1918] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:25.538722 8058 log.cc:1196] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/99eae7e5f3bc464397e838605348629e
I20260811 12:33:25.539314 8058 ts_tablet_manager.cc:1939] T 99eae7e5f3bc464397e838605348629e P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting consensus metadata
I20260811 12:33:25.541174 7091 catalog_manager.cc:4841] TS 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973): tablet 99eae7e5f3bc464397e838605348629e (table test-table [id=e2f9846c2baf4aa297216a128cf4b115]) successfully deleted
I20260811 12:33:25.542346 7980 ts_tablet_manager.cc:1428] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260811 12:33:25.542786 7981 consensus_queue.cc:260] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.544566 7981 ts_tablet_manager.cc:1428] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20260811 12:33:25.875583 7988 raft_consensus.cc:482] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:25.875854 7988 raft_consensus.cc:504] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.877457 7988 leader_election.cc:288] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579), 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:25.886049 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.878130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:40899 (local address 127.3.204.67:40973)
0811 12:33:25.878318 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.878324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.879093 (+ 769us) server_negotiation.cc:407] Connection header received
0811 12:33:25.879361 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.879366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.879445 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.879508 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:25.880394 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.881504 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.883765 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.885046 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.885224 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.885310 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.885614 (+ 304us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.885757 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:33:25.886193 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.877726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:40973 (local address 127.3.204.65:40899)
0811 12:33:25.879001 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:25.879017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:25.879191 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:25.879721 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:25.879725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:25.879747 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:25.880222 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.880235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.881651 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.881659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.883591 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.883607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.885226 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.885232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.885419 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.885461 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 12:33:25.885569 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 12:33:25.885717 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":36,"thread_start_us":663,"threads_started":1}
I20260811 12:33:25.886536 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.877686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46579 (local address 127.3.204.65:35409)
0811 12:33:25.878636 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:25.878664 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:25.878836 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:25.879583 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:25.879588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:25.879614 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:25.879963 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.879975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.881598 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.881603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.883870 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:25.883890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.885485 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.885492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:25.885674 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.885708 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 12:33:25.885803 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 12:33:25.885942 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":78,"thread_start_us":1022,"threads_started":1}
I20260811 12:33:25.886893 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.878483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:35409 (local address 127.3.204.66:46579)
0811 12:33:25.879260 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.879265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.879295 (+ 30us) server_negotiation.cc:407] Connection header received
0811 12:33:25.879360 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.879365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.879439 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.880190 (+ 751us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:25.880365 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.881432 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.884060 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.885243 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.885413 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.885482 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.886474 (+ 992us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.886630 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":549,"threads_started":1}
I20260811 12:33:25.887125 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e174fbc1eff54166999dd34abb0c5cda" candidate_uuid: "02b2237deace4028b90bea06f6b4af67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" is_pre_election: true
I20260811 12:33:25.887461 7857 raft_consensus.cc:2454] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b2237deace4028b90bea06f6b4af67 in term 0.
I20260811 12:33:25.887531 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e174fbc1eff54166999dd34abb0c5cda" candidate_uuid: "02b2237deace4028b90bea06f6b4af67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b7060a4c9247b6b6d6122fef7b0092" is_pre_election: true
I20260811 12:33:25.887784 7632 raft_consensus.cc:2454] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b2237deace4028b90bea06f6b4af67 in term 0.
I20260811 12:33:25.888108 7261 leader_election.cc:302] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67, 63ad2cd4cdc043f4a2b131e636497d2d; no voters:
I20260811 12:33:25.888549 7988 raft_consensus.cc:2781] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:25.888689 7988 raft_consensus.cc:482] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:25.888861 7988 raft_consensus.cc:3037] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:25.892314 7988 raft_consensus.cc:504] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.893049 7988 leader_election.cc:288] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [CANDIDATE]: Term 1 election: Requested vote from peers 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579), 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:25.893679 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e174fbc1eff54166999dd34abb0c5cda" candidate_uuid: "02b2237deace4028b90bea06f6b4af67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b7060a4c9247b6b6d6122fef7b0092"
I20260811 12:33:25.893679 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e174fbc1eff54166999dd34abb0c5cda" candidate_uuid: "02b2237deace4028b90bea06f6b4af67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ad2cd4cdc043f4a2b131e636497d2d"
I20260811 12:33:25.893910 7857 raft_consensus.cc:3037] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:25.893923 7632 raft_consensus.cc:3037] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:25.896332 7632 raft_consensus.cc:2454] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b2237deace4028b90bea06f6b4af67 in term 1.
I20260811 12:33:25.896890 7259 leader_election.cc:302] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67, 23b7060a4c9247b6b6d6122fef7b0092; no voters:
I20260811 12:33:25.897298 7988 raft_consensus.cc:2781] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:25.897529 7857 raft_consensus.cc:2454] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b2237deace4028b90bea06f6b4af67 in term 1.
I20260811 12:33:25.898696 7988 raft_consensus.cc:686] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 1 LEADER]: Becoming Leader. State: Replica: 02b2237deace4028b90bea06f6b4af67, State: Running, Role: LEADER
I20260811 12:33:25.899119 7988 consensus_queue.cc:237] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:25.903154 7112 catalog_manager.cc:5495] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02b2237deace4028b90bea06f6b4af67 (127.3.204.65). New cstate: current_term: 1 leader_uuid: "02b2237deace4028b90bea06f6b4af67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } health_report { overall_health: UNKNOWN } } }
I20260811 12:33:25.990654 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:25.984767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48406 (local address 127.3.204.65:34373)
0811 12:33:25.985327 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:25.985331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:25.985354 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:33:25.985543 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:25.985547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:25.985626 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:25.985766 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:25.986117 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.986885 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.988275 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:25.988784 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:25.988880 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:25.989271 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:25.990000 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:25.990141 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:25.990182 (+ 41us) server_negotiation.cc:299] Negotiation successful
0811 12:33:25.990249 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":326,"threads_started":1}
I20260811 12:33:26.006525 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:26.001059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:54927 (local address 127.3.204.66:46579)
0811 12:33:26.001296 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:26.001307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:26.001654 (+ 347us) server_negotiation.cc:407] Connection header received
0811 12:33:26.001849 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:26.001854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:26.001923 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:26.002003 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:26.002538 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:26.003409 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:26.004852 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:26.005268 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:26.005383 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:26.005818 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:26.005923 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:26.006000 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:26.006133 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 12:33:26.006214 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 12:33:26.006611 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:26.000891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46579 (local address 127.3.204.65:54927)
0811 12:33:26.001543 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:26.001563 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:26.001714 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:26.002035 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:26.002040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:26.002053 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:26.002359 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:26.002367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:26.003559 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:26.003563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:26.004690 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:26.004704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:26.005455 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:26.005461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:26.005582 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:26.005638 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:26.006020 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:26.006032 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 12:33:26.006101 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 12:33:26.006253 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":295,"threads_started":1}
I20260811 12:33:26.015673 7857 raft_consensus.cc:1261] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Refusing update from remote peer 02b2237deace4028b90bea06f6b4af67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:26.015673 7632 raft_consensus.cc:1261] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Refusing update from remote peer 02b2237deace4028b90bea06f6b4af67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:26.016448 8074 consensus_queue.cc:1035] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:26.016830 7988 consensus_queue.cc:1035] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:26.045104 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:26.039229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:47937 (local address 127.3.204.65:34373)
0811 12:33:26.039384 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:26.039391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:26.039909 (+ 518us) server_negotiation.cc:407] Connection header received
0811 12:33:26.040116 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:26.040121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:26.040204 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:26.040297 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:26.040751 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:26.041631 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:26.043400 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:26.043807 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:26.043939 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:26.044363 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:26.044469 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:26.044546 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:26.044697 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 12:33:26.044780 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 12:33:26.045179 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:26.039179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34373 (local address 127.3.204.66:47937)
0811 12:33:26.039804 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:26.039826 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:26.039963 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:26.040318 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:26.040323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:26.040337 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:26.040607 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:26.040622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:26.041751 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:26.041755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:26.043255 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:26.043269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:26.043946 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:26.043952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:26.044161 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:26.044212 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:26.044562 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:26.044574 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 12:33:26.044646 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 12:33:26.044789 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":285,"threads_started":1}
W20260811 12:33:26.046167 8081 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 12:33:27.649169 7108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:27.649425 7108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:27.653764 7108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: Sending DeleteTablet for 3 replicas of tablet e174fbc1eff54166999dd34abb0c5cda
I20260811 12:33:27.654656 7335 tablet_service.cc:1514] Processing DeleteTablet for tablet e174fbc1eff54166999dd34abb0c5cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:27 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:48392
I20260811 12:33:27.654727 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet e174fbc1eff54166999dd34abb0c5cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:27 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:45526
I20260811 12:33:27.654770 7829 tablet_service.cc:1514] Processing DeleteTablet for tablet e174fbc1eff54166999dd34abb0c5cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:27 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:59332
I20260811 12:33:27.667205 8151 tablet_replica.cc:330] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: stopping tablet replica
I20260811 12:33:27.667605 8151 raft_consensus.cc:2229] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:33:27.668401 8151 raft_consensus.cc:2258] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:27.669121 8152 tablet_replica.cc:330] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: stopping tablet replica
I20260811 12:33:27.669509 8155 tablet_replica.cc:330] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: stopping tablet replica
I20260811 12:33:27.669703 8155 raft_consensus.cc:2229] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:27.669936 8155 raft_consensus.cc:2258] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:27.669945 8151 ts_tablet_manager.cc:1905] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:27.670014 8152 raft_consensus.cc:2229] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:27.670280 8152 raft_consensus.cc:2258] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:27.671231 8152 ts_tablet_manager.cc:1905] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:27.671500 8155 ts_tablet_manager.cc:1905] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:27.676258 8151 ts_tablet_manager.cc:1918] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 12:33:27.676460 8151 log.cc:1196] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/e174fbc1eff54166999dd34abb0c5cda
I20260811 12:33:27.677027 8151 ts_tablet_manager.cc:1939] T e174fbc1eff54166999dd34abb0c5cda P 02b2237deace4028b90bea06f6b4af67: Deleting consensus metadata
I20260811 12:33:27.677755 8152 ts_tablet_manager.cc:1918] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 12:33:27.677980 8152 log.cc:1196] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/e174fbc1eff54166999dd34abb0c5cda
I20260811 12:33:27.678370 8155 ts_tablet_manager.cc:1918] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 12:33:27.678421 7091 catalog_manager.cc:4841] TS 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373): tablet e174fbc1eff54166999dd34abb0c5cda (table test-table [id=e275e4b9885145e69e3515503ceaff2a]) successfully deleted
I20260811 12:33:27.678522 8152 ts_tablet_manager.cc:1939] T e174fbc1eff54166999dd34abb0c5cda P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting consensus metadata
I20260811 12:33:27.678560 8155 log.cc:1196] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/e174fbc1eff54166999dd34abb0c5cda
I20260811 12:33:27.678972 8155 ts_tablet_manager.cc:1939] T e174fbc1eff54166999dd34abb0c5cda P 23b7060a4c9247b6b6d6122fef7b0092: Deleting consensus metadata
I20260811 12:33:27.679662 7091 catalog_manager.cc:4841] TS 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973): tablet e174fbc1eff54166999dd34abb0c5cda (table test-table [id=e275e4b9885145e69e3515503ceaff2a]) successfully deleted
I20260811 12:33:27.680128 7089 catalog_manager.cc:4841] TS 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579): tablet e174fbc1eff54166999dd34abb0c5cda (table test-table [id=e275e4b9885145e69e3515503ceaff2a]) successfully deleted
Aug 11 12:33:27 dist-test-slave-lgr5 krb5kdc[7033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 12:33:27 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:27.718626 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:27.703452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.3.204.126:35433)
0811 12:33:27.704015 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:27.704020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:27.704161 (+ 141us) server_negotiation.cc:407] Connection header received
0811 12:33:27.704323 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:27.704327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:27.704433 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:27.704538 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:27.705791 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:27.706583 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:27.707787 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:27.708248 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:27.708349 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:27.712531 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:27.712537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:27.712540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:27.712578 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:27.716053 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:27.716692 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:27.716698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:27.716702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:27.716877 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:27.717236 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:27.717240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:27.717243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:27.717604 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:27.717852 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:27.718008 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 12:33:27.718240 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":295,"threads_started":1}
I20260811 12:33:27.722373 7108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:27.723070 7108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:27.729938 7335 tablet_service.cc:1467] Processing CreateTablet for tablet 1debed26f9d34c5e9c838a1d86bd3897 (DEFAULT_TABLE table=test-table [id=d0ae1bced95f42668ebb2c2438c2a6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:27.730458 7335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1debed26f9d34c5e9c838a1d86bd3897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:27.730396 7601 tablet_service.cc:1467] Processing CreateTablet for tablet 1debed26f9d34c5e9c838a1d86bd3897 (DEFAULT_TABLE table=test-table [id=d0ae1bced95f42668ebb2c2438c2a6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:27.730973 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1debed26f9d34c5e9c838a1d86bd3897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:27.735920 7829 tablet_service.cc:1467] Processing CreateTablet for tablet 1debed26f9d34c5e9c838a1d86bd3897 (DEFAULT_TABLE table=test-table [id=d0ae1bced95f42668ebb2c2438c2a6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:27.736426 7829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1debed26f9d34c5e9c838a1d86bd3897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:27.738086 8168 tablet_bootstrap.cc:492] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: Bootstrap starting.
I20260811 12:33:27.740588 8168 tablet_bootstrap.cc:654] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:27.742065 8172 tablet_bootstrap.cc:492] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: Bootstrap starting.
I20260811 12:33:27.743223 8167 tablet_bootstrap.cc:492] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: Bootstrap starting.
I20260811 12:33:27.744720 8168 tablet_bootstrap.cc:492] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: No bootstrap required, opened a new log
I20260811 12:33:27.744881 8168 ts_tablet_manager.cc:1397] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 12:33:27.744944 8172 tablet_bootstrap.cc:654] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:27.745574 8168 raft_consensus.cc:348] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.745795 8168 raft_consensus.cc:374] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:27.745860 8167 tablet_bootstrap.cc:654] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:27.745913 8168 raft_consensus.cc:729] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b2237deace4028b90bea06f6b4af67, State: Initialized, Role: FOLLOWER
I20260811 12:33:27.746174 8168 consensus_queue.cc:260] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.746806 8168 ts_tablet_manager.cc:1428] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 12:33:27.748296 8172 tablet_bootstrap.cc:492] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: No bootstrap required, opened a new log
I20260811 12:33:27.748449 8172 ts_tablet_manager.cc:1397] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260811 12:33:27.748483 8167 tablet_bootstrap.cc:492] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: No bootstrap required, opened a new log
I20260811 12:33:27.748624 8167 ts_tablet_manager.cc:1397] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 12:33:27.749250 8172 raft_consensus.cc:348] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.749346 8167 raft_consensus.cc:348] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.749480 8172 raft_consensus.cc:374] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:27.749531 8167 raft_consensus.cc:374] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:27.749593 8167 raft_consensus.cc:729] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b7060a4c9247b6b6d6122fef7b0092, State: Initialized, Role: FOLLOWER
I20260811 12:33:27.749580 8172 raft_consensus.cc:729] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ad2cd4cdc043f4a2b131e636497d2d, State: Initialized, Role: FOLLOWER
I20260811 12:33:27.749804 8167 consensus_queue.cc:260] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.749811 8172 consensus_queue.cc:260] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.750322 8167 ts_tablet_manager.cc:1428] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260811 12:33:27.751433 8172 ts_tablet_manager.cc:1428] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:27.893654 8114 raft_consensus.cc:482] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:27.893889 8114 raft_consensus.cc:504] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.894562 8114 leader_election.cc:288] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373), 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:27.895373 7388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1debed26f9d34c5e9c838a1d86bd3897" candidate_uuid: "23b7060a4c9247b6b6d6122fef7b0092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b2237deace4028b90bea06f6b4af67" is_pre_election: true
I20260811 12:33:27.895435 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1debed26f9d34c5e9c838a1d86bd3897" candidate_uuid: "23b7060a4c9247b6b6d6122fef7b0092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" is_pre_election: true
I20260811 12:33:27.895707 7388 raft_consensus.cc:2454] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b7060a4c9247b6b6d6122fef7b0092 in term 0.
I20260811 12:33:27.895759 7855 raft_consensus.cc:2454] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b7060a4c9247b6b6d6122fef7b0092 in term 0.
I20260811 12:33:27.896309 7512 leader_election.cc:302] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67, 23b7060a4c9247b6b6d6122fef7b0092; no voters:
I20260811 12:33:27.896572 8114 raft_consensus.cc:2781] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:27.896704 8114 raft_consensus.cc:482] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:27.896800 8114 raft_consensus.cc:3037] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:27.899107 8114 raft_consensus.cc:504] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.899628 8114 leader_election.cc:288] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [CANDIDATE]: Term 1 election: Requested vote from peers 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373), 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:27.900266 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1debed26f9d34c5e9c838a1d86bd3897" candidate_uuid: "23b7060a4c9247b6b6d6122fef7b0092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ad2cd4cdc043f4a2b131e636497d2d"
I20260811 12:33:27.900267 7388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1debed26f9d34c5e9c838a1d86bd3897" candidate_uuid: "23b7060a4c9247b6b6d6122fef7b0092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b2237deace4028b90bea06f6b4af67"
I20260811 12:33:27.900501 7388 raft_consensus.cc:3037] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:27.900521 7855 raft_consensus.cc:3037] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:27.903461 7388 raft_consensus.cc:2454] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b7060a4c9247b6b6d6122fef7b0092 in term 1.
I20260811 12:33:27.903471 7855 raft_consensus.cc:2454] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b7060a4c9247b6b6d6122fef7b0092 in term 1.
I20260811 12:33:27.903887 7512 leader_election.cc:302] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67, 23b7060a4c9247b6b6d6122fef7b0092; no voters:
I20260811 12:33:27.904215 8114 raft_consensus.cc:2781] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:27.904347 8114 raft_consensus.cc:686] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 LEADER]: Becoming Leader. State: Replica: 23b7060a4c9247b6b6d6122fef7b0092, State: Running, Role: LEADER
I20260811 12:33:27.904695 8114 consensus_queue.cc:237] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:27.907564 7112 catalog_manager.cc:5495] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66). New cstate: current_term: 1 leader_uuid: "23b7060a4c9247b6b6d6122fef7b0092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } health_report { overall_health: UNKNOWN } } }
I20260811 12:33:27.939909 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:27.934676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.3.204.66:46579)
0811 12:33:27.935210 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:27.935215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:27.935234 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:33:27.935287 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:27.935290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:27.935348 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:27.935436 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:27.935822 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:27.936638 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:27.938049 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:27.938518 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:27.938624 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:27.939006 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:27.939122 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:27.939187 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:27.939278 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 12:33:27.939343 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":277,"threads_started":1}
I20260811 12:33:27.953752 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:27.948759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:58287 (local address 127.3.204.66:46579)
0811 12:33:27.948975 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:27.948985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:27.949177 (+ 192us) server_negotiation.cc:407] Connection header received
0811 12:33:27.949346 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:27.949351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:27.949417 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:27.949489 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:27.949956 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:27.950743 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:27.952100 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:27.952516 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:27.952622 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:27.953041 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:27.953142 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:27.953213 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:27.953337 (+ 124us) server_negotiation.cc:299] Negotiation successful
0811 12:33:27.953406 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:33:27.953801 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:27.948568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46579 (local address 127.3.204.66:58287)
0811 12:33:27.949088 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:27.949103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:27.949236 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:27.949512 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:27.949516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:27.949527 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:33:27.949781 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:27.949790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:27.950885 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:27.950891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:27.951949 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:27.951963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:27.952669 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:27.952676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:27.952854 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:27.952896 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:33:27.953226 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:33:27.953237 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 12:33:27.953310 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 12:33:27.953449 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":270,"threads_started":1}
I20260811 12:33:27.961711 7855 raft_consensus.cc:1261] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Refusing update from remote peer 23b7060a4c9247b6b6d6122fef7b0092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:27.961776 7388 raft_consensus.cc:1261] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Refusing update from remote peer 23b7060a4c9247b6b6d6122fef7b0092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:27.962319 8122 consensus_queue.cc:1035] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:27.962656 8114 consensus_queue.cc:1035] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:28.012045 7112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:28.012351 7112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:28.015434 7112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: Sending DeleteTablet for 3 replicas of tablet 1debed26f9d34c5e9c838a1d86bd3897
I20260811 12:33:28.016363 7335 tablet_service.cc:1514] Processing DeleteTablet for tablet 1debed26f9d34c5e9c838a1d86bd3897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:28 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:48392
I20260811 12:33:28.016675 8151 tablet_replica.cc:330] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: stopping tablet replica
I20260811 12:33:28.016825 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet 1debed26f9d34c5e9c838a1d86bd3897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:28 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:45526
I20260811 12:33:28.016878 8151 raft_consensus.cc:2229] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:28.017100 8151 raft_consensus.cc:2258] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:28.017217 8155 tablet_replica.cc:330] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: stopping tablet replica
I20260811 12:33:28.017441 8155 raft_consensus.cc:2229] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:33:28.017905 7829 tablet_service.cc:1514] Processing DeleteTablet for tablet 1debed26f9d34c5e9c838a1d86bd3897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:28 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:59332
I20260811 12:33:28.018021 8155 raft_consensus.cc:2258] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:28.018218 8152 tablet_replica.cc:330] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: stopping tablet replica
I20260811 12:33:28.018270 8151 ts_tablet_manager.cc:1905] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:28.018411 8152 raft_consensus.cc:2229] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:28.018626 8152 raft_consensus.cc:2258] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:28.019265 8155 ts_tablet_manager.cc:1905] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:28.019311 7112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:28.020012 7112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:28.022029 8152 ts_tablet_manager.cc:1905] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:28.025061 8151 ts_tablet_manager.cc:1918] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:28.027282 8151 log.cc:1196] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/1debed26f9d34c5e9c838a1d86bd3897
I20260811 12:33:28.027408 8155 ts_tablet_manager.cc:1918] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:28.027587 8155 log.cc:1196] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/1debed26f9d34c5e9c838a1d86bd3897
I20260811 12:33:28.027696 8151 ts_tablet_manager.cc:1939] T 1debed26f9d34c5e9c838a1d86bd3897 P 02b2237deace4028b90bea06f6b4af67: Deleting consensus metadata
I20260811 12:33:28.027983 8155 ts_tablet_manager.cc:1939] T 1debed26f9d34c5e9c838a1d86bd3897 P 23b7060a4c9247b6b6d6122fef7b0092: Deleting consensus metadata
I20260811 12:33:28.028863 7091 catalog_manager.cc:4841] TS 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373): tablet 1debed26f9d34c5e9c838a1d86bd3897 (table test-table [id=d0ae1bced95f42668ebb2c2438c2a6f7]) successfully deleted
I20260811 12:33:28.028640 8152 ts_tablet_manager.cc:1918] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:33:28.029220 8152 log.cc:1196] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/1debed26f9d34c5e9c838a1d86bd3897
I20260811 12:33:28.029662 8152 ts_tablet_manager.cc:1939] T 1debed26f9d34c5e9c838a1d86bd3897 P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting consensus metadata
I20260811 12:33:28.030061 7089 catalog_manager.cc:4841] TS 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579): tablet 1debed26f9d34c5e9c838a1d86bd3897 (table test-table [id=d0ae1bced95f42668ebb2c2438c2a6f7]) successfully deleted
I20260811 12:33:28.030679 7091 catalog_manager.cc:4841] TS 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973): tablet 1debed26f9d34c5e9c838a1d86bd3897 (table test-table [id=d0ae1bced95f42668ebb2c2438c2a6f7]) successfully deleted
I20260811 12:33:28.031615 7335 tablet_service.cc:1467] Processing CreateTablet for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 (DEFAULT_TABLE table=test-table [id=8552ef437d2d4e6dbf6647a1fb15c63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:28.031602 7601 tablet_service.cc:1467] Processing CreateTablet for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 (DEFAULT_TABLE table=test-table [id=8552ef437d2d4e6dbf6647a1fb15c63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:28.031811 7829 tablet_service.cc:1467] Processing CreateTablet for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 (DEFAULT_TABLE table=test-table [id=8552ef437d2d4e6dbf6647a1fb15c63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:28.032107 7335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:28.032279 7829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:28.032960 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:28.038545 8168 tablet_bootstrap.cc:492] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: Bootstrap starting.
I20260811 12:33:28.038982 8172 tablet_bootstrap.cc:492] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: Bootstrap starting.
I20260811 12:33:28.040277 8168 tablet_bootstrap.cc:654] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:28.040783 8172 tablet_bootstrap.cc:654] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:28.043496 8167 tablet_bootstrap.cc:492] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: Bootstrap starting.
I20260811 12:33:28.045459 8167 tablet_bootstrap.cc:654] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:28.054613 8168 tablet_bootstrap.cc:492] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: No bootstrap required, opened a new log
I20260811 12:33:28.054872 8168 ts_tablet_manager.cc:1397] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.011s
I20260811 12:33:28.055285 8167 tablet_bootstrap.cc:492] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: No bootstrap required, opened a new log
I20260811 12:33:28.055505 8167 ts_tablet_manager.cc:1397] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260811 12:33:28.055841 8168 raft_consensus.cc:348] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.056166 8168 raft_consensus.cc:374] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:28.056243 8168 raft_consensus.cc:729] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b2237deace4028b90bea06f6b4af67, State: Initialized, Role: FOLLOWER
I20260811 12:33:28.056417 8167 raft_consensus.cc:348] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.056519 8168 consensus_queue.cc:260] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.056741 8167 raft_consensus.cc:374] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:28.056847 8167 raft_consensus.cc:729] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b7060a4c9247b6b6d6122fef7b0092, State: Initialized, Role: FOLLOWER
I20260811 12:33:28.057133 8167 consensus_queue.cc:260] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.059698 8168 ts_tablet_manager.cc:1428] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260811 12:33:28.060014 8167 ts_tablet_manager.cc:1428] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 12:33:28.063771 8172 tablet_bootstrap.cc:492] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: No bootstrap required, opened a new log
I20260811 12:33:28.064114 8172 ts_tablet_manager.cc:1397] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.007s
I20260811 12:33:28.065753 8172 raft_consensus.cc:348] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.066110 8172 raft_consensus.cc:374] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:28.066188 8172 raft_consensus.cc:729] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ad2cd4cdc043f4a2b131e636497d2d, State: Initialized, Role: FOLLOWER
I20260811 12:33:28.066535 8172 consensus_queue.cc:260] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.067085 8172 ts_tablet_manager.cc:1428] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260811 12:33:28.146031 8178 raft_consensus.cc:482] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:28.146297 8178 raft_consensus.cc:504] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.147218 8178 leader_election.cc:288] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373), 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579)
I20260811 12:33:28.147884 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a746e1d8eb4ad8ba6a40443a30bbf2" candidate_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b7060a4c9247b6b6d6122fef7b0092" is_pre_election: true
I20260811 12:33:28.147884 7387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a746e1d8eb4ad8ba6a40443a30bbf2" candidate_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b2237deace4028b90bea06f6b4af67" is_pre_election: true
I20260811 12:33:28.148111 8137 raft_consensus.cc:482] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:28.148213 7632 raft_consensus.cc:2454] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ad2cd4cdc043f4a2b131e636497d2d in term 0.
I20260811 12:33:28.148272 8137 raft_consensus.cc:504] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.148676 7387 raft_consensus.cc:2454] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63ad2cd4cdc043f4a2b131e636497d2d in term 0.
I20260811 12:33:28.148793 7754 leader_election.cc:302] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b7060a4c9247b6b6d6122fef7b0092, 63ad2cd4cdc043f4a2b131e636497d2d; no voters:
I20260811 12:33:28.149004 8137 leader_election.cc:288] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579), 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:28.149590 8178 raft_consensus.cc:2781] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:28.149739 8178 raft_consensus.cc:482] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:28.149680 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a746e1d8eb4ad8ba6a40443a30bbf2" candidate_uuid: "02b2237deace4028b90bea06f6b4af67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b7060a4c9247b6b6d6122fef7b0092" is_pre_election: true
I20260811 12:33:28.149837 8178 raft_consensus.cc:3037] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:28.149767 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a746e1d8eb4ad8ba6a40443a30bbf2" candidate_uuid: "02b2237deace4028b90bea06f6b4af67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" is_pre_election: true
I20260811 12:33:28.149915 7632 raft_consensus.cc:2454] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b2237deace4028b90bea06f6b4af67 in term 0.
I20260811 12:33:28.150357 7259 leader_election.cc:302] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67, 23b7060a4c9247b6b6d6122fef7b0092; no voters:
I20260811 12:33:28.150638 8137 raft_consensus.cc:2781] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:28.150768 8137 raft_consensus.cc:482] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:28.150978 8137 raft_consensus.cc:3037] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:28.152494 8178 raft_consensus.cc:504] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.153046 8178 leader_election.cc:288] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [CANDIDATE]: Term 1 election: Requested vote from peers 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373), 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579)
I20260811 12:33:28.153684 7387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a746e1d8eb4ad8ba6a40443a30bbf2" candidate_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b2237deace4028b90bea06f6b4af67"
I20260811 12:33:28.153789 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a746e1d8eb4ad8ba6a40443a30bbf2" candidate_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b7060a4c9247b6b6d6122fef7b0092"
I20260811 12:33:28.153981 7632 raft_consensus.cc:3037] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:28.154234 8137 raft_consensus.cc:504] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.154685 7387 raft_consensus.cc:2379] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 63ad2cd4cdc043f4a2b131e636497d2d in current term 1: Already voted for candidate 02b2237deace4028b90bea06f6b4af67 in this term.
I20260811 12:33:28.155144 7855 raft_consensus.cc:2379] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02b2237deace4028b90bea06f6b4af67 in current term 1: Already voted for candidate 63ad2cd4cdc043f4a2b131e636497d2d in this term.
I20260811 12:33:28.156554 8137 leader_election.cc:288] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [CANDIDATE]: Term 1 election: Requested vote from peers 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579), 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973)
I20260811 12:33:28.156629 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a746e1d8eb4ad8ba6a40443a30bbf2" candidate_uuid: "02b2237deace4028b90bea06f6b4af67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b7060a4c9247b6b6d6122fef7b0092"
I20260811 12:33:28.156754 7632 raft_consensus.cc:2454] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ad2cd4cdc043f4a2b131e636497d2d in term 1.
I20260811 12:33:28.157181 7754 leader_election.cc:302] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b7060a4c9247b6b6d6122fef7b0092, 63ad2cd4cdc043f4a2b131e636497d2d; no voters: 02b2237deace4028b90bea06f6b4af67
I20260811 12:33:28.157546 8178 raft_consensus.cc:2781] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:28.157653 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a746e1d8eb4ad8ba6a40443a30bbf2" candidate_uuid: "02b2237deace4028b90bea06f6b4af67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ad2cd4cdc043f4a2b131e636497d2d"
I20260811 12:33:28.158051 8178 raft_consensus.cc:686] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 LEADER]: Becoming Leader. State: Replica: 63ad2cd4cdc043f4a2b131e636497d2d, State: Running, Role: LEADER
I20260811 12:33:28.158347 8178 consensus_queue.cc:237] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } }
I20260811 12:33:28.160283 7261 leader_election.cc:302] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02b2237deace4028b90bea06f6b4af67; no voters: 23b7060a4c9247b6b6d6122fef7b0092, 63ad2cd4cdc043f4a2b131e636497d2d
I20260811 12:33:28.160701 8137 raft_consensus.cc:2726] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260811 12:33:28.161393 7112 catalog_manager.cc:5495] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67). New cstate: current_term: 1 leader_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63ad2cd4cdc043f4a2b131e636497d2d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 40973 } health_report { overall_health: HEALTHY } } }
I20260811 12:33:28.185276 8200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:28.178907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.3.204.67:40973)
0811 12:33:28.179551 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:28.179558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:28.179590 (+ 32us) server_negotiation.cc:407] Connection header received
0811 12:33:28.179659 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:28.179665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:28.179747 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:28.179856 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:28.180334 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:28.181608 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:28.183049 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:28.183777 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:28.183906 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:28.184450 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:28.184605 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:28.184700 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:28.184789 (+ 89us) server_negotiation.cc:299] Negotiation successful
0811 12:33:28.184881 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":565,"threads_started":1}
I20260811 12:33:28.195564 7387 raft_consensus.cc:1261] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Refusing update from remote peer 63ad2cd4cdc043f4a2b131e636497d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:28.195876 7632 raft_consensus.cc:1261] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Refusing update from remote peer 63ad2cd4cdc043f4a2b131e636497d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:28.196452 8178 consensus_queue.cc:1035] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b2237deace4028b90bea06f6b4af67" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:28.196743 8198 consensus_queue.cc:1035] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b7060a4c9247b6b6d6122fef7b0092" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 12:33:28.223591 8123 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 12:33:29.844193 7110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:29.844424 7110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:29.848673 7110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f83fef6e626344cead90218557fb9ad1: Sending DeleteTablet for 3 replicas of tablet f5a746e1d8eb4ad8ba6a40443a30bbf2
I20260811 12:33:29.849537 7335 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:29 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:48392
I20260811 12:33:29.849556 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:29 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:45526
I20260811 12:33:29.849572 7829 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:29 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:59332
I20260811 12:33:29.850399 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7228
I20260811 12:33:29.862917 8257 tablet_replica.cc:330] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: stopping tablet replica
I20260811 12:33:29.863382 8257 raft_consensus.cc:2229] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:29.863794 8257 raft_consensus.cc:2258] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:29.865242 8257 ts_tablet_manager.cc:1905] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:29.866325 8252 tablet_replica.cc:330] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: stopping tablet replica
I20260811 12:33:29.866564 8252 raft_consensus.cc:2229] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:29.866837 8252 raft_consensus.cc:2258] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:29.868053 8256 tablet_replica.cc:330] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: stopping tablet replica
I20260811 12:33:29.868207 8252 ts_tablet_manager.cc:1905] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:29.868348 8256 raft_consensus.cc:2229] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:33:29.868760 8256 raft_consensus.cc:2258] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:29.869741 8256 ts_tablet_manager.cc:1905] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:29.874244 8257 ts_tablet_manager.cc:1918] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 12:33:29.874480 8257 log.cc:1196] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/f5a746e1d8eb4ad8ba6a40443a30bbf2
I20260811 12:33:29.875028 8257 ts_tablet_manager.cc:1939] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 23b7060a4c9247b6b6d6122fef7b0092: Deleting consensus metadata
I20260811 12:33:29.875854 8252 ts_tablet_manager.cc:1918] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 12:33:29.876001 8252 log.cc:1196] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/f5a746e1d8eb4ad8ba6a40443a30bbf2
I20260811 12:33:29.876334 8256 ts_tablet_manager.cc:1918] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 12:33:29.876518 8256 log.cc:1196] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/f5a746e1d8eb4ad8ba6a40443a30bbf2
I20260811 12:33:29.876513 7089 catalog_manager.cc:4841] TS 23b7060a4c9247b6b6d6122fef7b0092 (127.3.204.66:46579): tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 (table test-table [id=8552ef437d2d4e6dbf6647a1fb15c63c]) successfully deleted
I20260811 12:33:29.876572 8252 ts_tablet_manager.cc:1939] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 02b2237deace4028b90bea06f6b4af67: Deleting consensus metadata
I20260811 12:33:29.876957 8256 ts_tablet_manager.cc:1939] T f5a746e1d8eb4ad8ba6a40443a30bbf2 P 63ad2cd4cdc043f4a2b131e636497d2d: Deleting consensus metadata
I20260811 12:33:29.877800 7091 catalog_manager.cc:4841] TS 02b2237deace4028b90bea06f6b4af67 (127.3.204.65:34373): tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 (table test-table [id=8552ef437d2d4e6dbf6647a1fb15c63c]) successfully deleted
I20260811 12:33:29.878096 7091 catalog_manager.cc:4841] TS 63ad2cd4cdc043f4a2b131e636497d2d (127.3.204.67:40973): tablet f5a746e1d8eb4ad8ba6a40443a30bbf2 (table test-table [id=8552ef437d2d4e6dbf6647a1fb15c63c]) successfully deleted
Aug 11 12:33:29 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:29.885313 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:29.852249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.3.204.65:34373)
0811 12:33:29.859946 (+ 7697us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:29.859954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:29.867614 (+ 7660us) server_negotiation.cc:407] Connection header received
0811 12:33:29.867932 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:29.867938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:29.868035 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:29.868147 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:29.870047 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:29.871320 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:29.873433 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:29.874190 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:29.874297 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:29.879583 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:29.879599 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:29.879617 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:29.879651 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:29.882324 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:29.883023 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:29.883032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:29.883035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:29.883173 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:29.883619 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:29.883623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:29.883626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:29.884003 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:29.884333 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:29.884625 (+ 292us) server_negotiation.cc:299] Negotiation successful
0811 12:33:29.884886 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7439,"thread_start_us":7253,"threads_started":1}
W20260811 12:33:29.886097 7459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53228
W20260811 12:33:29.886762 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7228: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:29.886967 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7228
I20260811 12:33:29.921710 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7476
Aug 11 12:33:29 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:33:29.937343 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:29.922503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.3.204.66:46579)
0811 12:33:29.923016 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:29.923021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:29.923036 (+ 15us) server_negotiation.cc:407] Connection header received
0811 12:33:29.923090 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:29.923094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:29.923153 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:29.923254 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:29.924664 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:29.925463 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:29.926742 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:29.927135 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:29.927248 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:29.931568 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:29.931586 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:29.931603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:29.931648 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:29.934484 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:29.935182 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:29.935190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:29.935192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:29.935350 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:29.935707 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:29.935711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:29.935713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:29.936173 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:29.936507 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:29.936771 (+ 264us) server_negotiation.cc:299] Negotiation successful
0811 12:33:29.937004 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":238,"threads_started":1}
W20260811 12:33:29.938086 7714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36982
W20260811 12:33:29.938717 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7476: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:29.938867 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7476
I20260811 12:33:29.973138 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7727
Aug 11 12:33:29 dist-test-slave-lgr5 krb5kdc[7033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:33:29.990235 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:29.974178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.3.204.67:40973)
0811 12:33:29.974855 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:29.974861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:29.974881 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:33:29.974954 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:29.974960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:29.975053 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:29.975165 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:29.976919 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:29.978058 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:29.979696 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:29.980163 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:29.980279 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:29.984388 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:29.984405 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:29.984424 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:29.984467 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:29.987307 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:29.988012 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:29.988022 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:29.988026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:29.988215 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:29.988589 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:29.988594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:29.988598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:29.988993 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:29.989290 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:29.989598 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 12:33:29.989840 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":319,"threads_started":1}
W20260811 12:33:29.990995 7943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49858
W20260811 12:33:29.991629 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7727: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:29.991798 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7727
I20260811 12:33:30.024760 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7055
I20260811 12:33:30.025976 7183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:30.143077 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7055
2026-08-11T12:33:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8981 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:30 dist-test-slave-lgr5 krb5kdc[8270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:30 dist-test-slave-lgr5 krb5kdc[8270](info): set up 2 sockets
Aug 11 12:33:30 dist-test-slave-lgr5 krb5kdc[8270](info): commencing operation
krb5kdc: starting...
W20260811 12:33:31.373270 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:31 dist-test-slave-lgr5 krb5kdc[8270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:31.587229 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37535
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:37535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:31.919719 8289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:31.920032 8289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:31.920182 8289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:31.927492 8289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:31.927609 8289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:31.927680 8289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:31.927759 8289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:31.927819 8289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:31.937806 8289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:37535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:31.939189 8289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:31.940599 8289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:31.952543 8299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:31.953132 8296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:31.952613 8297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:31.953886 8289 server_base.cc:1034] running on GCE node
I20260811 12:33:31.954741 8289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:31.956266 8289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:31.957495 8289 hybrid_clock.cc:648] HybridClock initialized: now 1786451611957436 us; error 45 us; skew 500 ppm
I20260811 12:33:31.957865 8289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:31 dist-test-slave-lgr5 krb5kdc[8270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451611, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:31.960968 8289 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:31.962497 8289 webserver.cc:460] Webserver started at http://127.3.204.126:36693/ using document root <none> and password file <none>
I20260811 12:33:31.963222 8289 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:31.963341 8289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:31.963653 8289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:31.966785 8289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "312ee2f269d0426a84f64d708d7066b4"
format_stamp: "Formatted at 2026-08-11 12:33:31 on dist-test-slave-lgr5"
server_key: "da9aca06c2bdad24bad70984a7af35d3"
server_key_iv: "2a0d8283024b4b6ec0db6a899a047f1b"
server_key_version: "encryptionkey@0"
I20260811 12:33:31.967587 8289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "312ee2f269d0426a84f64d708d7066b4"
format_stamp: "Formatted at 2026-08-11 12:33:31 on dist-test-slave-lgr5"
server_key: "da9aca06c2bdad24bad70984a7af35d3"
server_key_iv: "2a0d8283024b4b6ec0db6a899a047f1b"
server_key_version: "encryptionkey@0"
I20260811 12:33:31.973395 8289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:33:31.983688 8310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:31.985514 8289 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.005s
I20260811 12:33:31.985705 8289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "312ee2f269d0426a84f64d708d7066b4"
format_stamp: "Formatted at 2026-08-11 12:33:31 on dist-test-slave-lgr5"
server_key: "da9aca06c2bdad24bad70984a7af35d3"
server_key_iv: "2a0d8283024b4b6ec0db6a899a047f1b"
server_key_version: "encryptionkey@0"
I20260811 12:33:31.985919 8289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:32.000785 8289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:32.005240 8289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:32.005534 8289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:32.023733 8289 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:37535
I20260811 12:33:32.023741 8410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:37535 every 8 connection(s)
I20260811 12:33:32.025207 8289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:32.029176 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:32.034587 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8289
I20260811 12:33:32.034823 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:32.035640 3889 external_mini_cluster.cc:1428] Setting key f0b0e02ce897870e90fd23ae8d851ff9
I20260811 12:33:32.041072 8411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4: Bootstrap starting.
I20260811 12:33:32.044356 8411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:32.045392 8411 log.cc:824] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:32.048511 8411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4: No bootstrap required, opened a new log
Aug 11 12:33:32 dist-test-slave-lgr5 krb5kdc[8270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:32.058683 8411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ee2f269d0426a84f64d708d7066b4" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37535 } }
I20260811 12:33:32.059194 8411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:32.059305 8411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312ee2f269d0426a84f64d708d7066b4, State: Initialized, Role: FOLLOWER
I20260811 12:33:32.060070 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:32.038024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.3.204.126:37535)
0811 12:33:32.038925 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:32.038938 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:32.038987 (+ 49us) server_negotiation.cc:407] Connection header received
0811 12:33:32.040772 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:32.040799 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:32.041234 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:32.041616 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:32.043184 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:32.044595 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:32.045980 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:32.046795 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:32.046986 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:32.051172 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:32.051185 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:32.051212 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:32.051261 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:32.055363 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:32.056036 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:32.056049 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:32.056059 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:32.056333 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:32.056673 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:32.056678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:32.056681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:32.057511 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:32.057819 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:32.058134 (+ 315us) server_negotiation.cc:299] Negotiation successful
0811 12:33:32.058613 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":419,"threads_started":1}
I20260811 12:33:32.060035 8411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ee2f269d0426a84f64d708d7066b4" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37535 } }
I20260811 12:33:32.060364 8411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:32.060497 8411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:32.060647 8411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:32.063012 8411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ee2f269d0426a84f64d708d7066b4" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37535 } }
I20260811 12:33:32.063510 8411 leader_election.cc:302] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 312ee2f269d0426a84f64d708d7066b4; no voters:
I20260811 12:33:32.064100 8411 leader_election.cc:288] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:32.064242 8423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:32.064922 8423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [term 1 LEADER]: Becoming Leader. State: Replica: 312ee2f269d0426a84f64d708d7066b4, State: Running, Role: LEADER
I20260811 12:33:32.065426 8423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ee2f269d0426a84f64d708d7066b4" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37535 } }
I20260811 12:33:32.066084 8411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:32.068243 8424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "312ee2f269d0426a84f64d708d7066b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ee2f269d0426a84f64d708d7066b4" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37535 } } }
I20260811 12:33:32.068629 8424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:32.069188 8425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 312ee2f269d0426a84f64d708d7066b4. Latest consensus state: current_term: 1 leader_uuid: "312ee2f269d0426a84f64d708d7066b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ee2f269d0426a84f64d708d7066b4" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37535 } } }
I20260811 12:33:32.069463 8425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:32.069571 8432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:32.075760 8432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:32.084563 8432 catalog_manager.cc:1347] Generated new cluster ID: 23eed0d0ca804ff89bae799832b0ba60
I20260811 12:33:32.084688 8432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:32.098429 8432 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:32.100455 8432 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:32.112170 8432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 312ee2f269d0426a84f64d708d7066b4: Generated new TSK 0
I20260811 12:33:32.113092 8432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:32.172616 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37535
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:32.512688 8457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:32.512992 8457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:32.513115 8457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:32.520453 8457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:32.520576 8457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:32.520744 8457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:32.530689 8457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:32.532110 8457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:32.533442 8457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:32.546669 8469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:32.546710 8470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:32.546635 8476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:32.547518 8457 server_base.cc:1034] running on GCE node
I20260811 12:33:32.548193 8457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:32.549168 8457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:32.550382 8457 hybrid_clock.cc:648] HybridClock initialized: now 1786451612550321 us; error 47 us; skew 500 ppm
I20260811 12:33:32.550709 8457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:32 dist-test-slave-lgr5 krb5kdc[8270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:32.553910 8457 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:32.555497 8457 webserver.cc:460] Webserver started at http://127.3.204.65:35435/ using document root <none> and password file <none>
I20260811 12:33:32.556269 8457 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:32.556394 8457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:32.556720 8457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:32.559729 8457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "adead27fa63944cf95251bbbe5c03a5f"
format_stamp: "Formatted at 2026-08-11 12:33:32 on dist-test-slave-lgr5"
server_key: "8f4fb5bf4f486715da498618e0aacf19"
server_key_iv: "11c2ded9210a671e8633334649260379"
server_key_version: "encryptionkey@0"
I20260811 12:33:32.560576 8457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "adead27fa63944cf95251bbbe5c03a5f"
format_stamp: "Formatted at 2026-08-11 12:33:32 on dist-test-slave-lgr5"
server_key: "8f4fb5bf4f486715da498618e0aacf19"
server_key_iv: "11c2ded9210a671e8633334649260379"
server_key_version: "encryptionkey@0"
I20260811 12:33:32.565971 8457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:33:32.569886 8489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:32.571285 8457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:33:32.571481 8457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "adead27fa63944cf95251bbbe5c03a5f"
format_stamp: "Formatted at 2026-08-11 12:33:32 on dist-test-slave-lgr5"
server_key: "8f4fb5bf4f486715da498618e0aacf19"
server_key_iv: "11c2ded9210a671e8633334649260379"
server_key_version: "encryptionkey@0"
I20260811 12:33:32.571671 8457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:32.596908 8457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:32.601153 8457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:32.601421 8457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:32.602470 8457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:32.603870 8457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:32.603943 8457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:32.604044 8457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:32.604130 8457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260811 12:33:32.634869 8457 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:40767
I20260811 12:33:32.634932 8692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:40767 every 8 connection(s)
I20260811 12:33:32.636185 8457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:32.646068 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8457
I20260811 12:33:32.646308 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:32.647087 3889 external_mini_cluster.cc:1428] Setting key a5659f9565624d3ff063ac32ca80e533
Aug 11 12:33:32 dist-test-slave-lgr5 krb5kdc[8270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:32.656440 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:32.638609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:37731 (local address 127.3.204.126:37535)
0811 12:33:32.639088 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:32.639096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:32.640030 (+ 934us) server_negotiation.cc:407] Connection header received
0811 12:33:32.641867 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:32.641872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:32.641946 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:32.642044 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:32.643572 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:32.644355 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:32.645866 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:32.646488 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:32.646632 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:32.650224 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:32.650231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:32.650237 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:32.650276 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:32.653351 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:32.654107 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:32.654113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:32.654116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:32.654260 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:32.654699 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:32.654703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:32.654705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:32.655038 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:32.655183 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:32.655864 (+ 681us) server_negotiation.cc:299] Negotiation successful
0811 12:33:32.656113 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":215,"threads_started":1}
I20260811 12:33:32.657624 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:32.639144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37535 (local address 127.3.204.65:37731)
0811 12:33:32.639860 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:32.639902 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:32.641593 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:32.642243 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:32.642251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:32.642754 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:32.643383 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:32.643401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:32.644498 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:32.644502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:32.645708 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:32.645720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:32.646704 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:32.646709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:32.646920 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:32.647575 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:32.647598 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:32.649981 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:32.653510 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:32.653517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:32.653531 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:32.653991 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:32.654394 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:32.654397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:32.654400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:32.654603 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:32.655150 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:32.655156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:32.655489 (+ 333us) client_negotiation.cc:769] Sending connection context
0811 12:33:32.655725 (+ 236us) client_negotiation.cc:240] Negotiation successful
0811 12:33:32.656196 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":268,"threads_started":1}
I20260811 12:33:32.659260 8697 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37535
I20260811 12:33:32.659611 8697 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:32.660552 8697 heartbeater.cc:511] Master 127.3.204.126:37535 requested a full tablet report, sending...
I20260811 12:33:32.662797 8346 ts_manager.cc:194] Registered new tserver with Master: adead27fa63944cf95251bbbe5c03a5f (127.3.204.65:40767)
I20260811 12:33:32.664307 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:37731
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:32.695389 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37535
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:33.031178 8707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:33.031541 8707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:33.031692 8707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:33.043183 8707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:33.043347 8707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:33.043550 8707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:33.054507 8707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:33.055856 8707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:33.057236 8707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:33.071349 8719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:33.071381 8712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:33.071313 8716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:33.072322 8707 server_base.cc:1034] running on GCE node
I20260811 12:33:33.073007 8707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:33.074002 8707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:33.075217 8707 hybrid_clock.cc:648] HybridClock initialized: now 1786451613075151 us; error 53 us; skew 500 ppm
I20260811 12:33:33.075536 8707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:33 dist-test-slave-lgr5 krb5kdc[8270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451613, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:33.078895 8707 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:33.080533 8707 webserver.cc:460] Webserver started at http://127.3.204.66:41635/ using document root <none> and password file <none>
I20260811 12:33:33.081367 8707 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:33.081499 8707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:33.081815 8707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:33.084980 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "93af4564829a442499a0f2cf9f087e1b"
format_stamp: "Formatted at 2026-08-11 12:33:33 on dist-test-slave-lgr5"
server_key: "d19bd00e61a91675efac34af6992644e"
server_key_iv: "94823634ade4010143beee6b6e1671a6"
server_key_version: "encryptionkey@0"
I20260811 12:33:33.085881 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "93af4564829a442499a0f2cf9f087e1b"
format_stamp: "Formatted at 2026-08-11 12:33:33 on dist-test-slave-lgr5"
server_key: "d19bd00e61a91675efac34af6992644e"
server_key_iv: "94823634ade4010143beee6b6e1671a6"
server_key_version: "encryptionkey@0"
I20260811 12:33:33.091688 8707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260811 12:33:33.095315 8729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:33.096781 8707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:33.096966 8707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "93af4564829a442499a0f2cf9f087e1b"
format_stamp: "Formatted at 2026-08-11 12:33:33 on dist-test-slave-lgr5"
server_key: "d19bd00e61a91675efac34af6992644e"
server_key_iv: "94823634ade4010143beee6b6e1671a6"
server_key_version: "encryptionkey@0"
I20260811 12:33:33.097186 8707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:33.126602 8707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:33.131160 8707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:33.131465 8707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:33.132555 8707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:33.134009 8707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:33.134089 8707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:33.134215 8707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:33.134271 8707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:33.166198 8707 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:34239
I20260811 12:33:33.166245 8944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:34239 every 8 connection(s)
I20260811 12:33:33.167550 8707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:33.177145 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8707
I20260811 12:33:33.177325 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:33.177790 3889 external_mini_cluster.cc:1428] Setting key fbb1fa244b833c5fc5861e8543b84e64
Aug 11 12:33:33 dist-test-slave-lgr5 krb5kdc[8270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451613, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:33.189599 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:33.170270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:45391 (local address 127.3.204.126:37535)
0811 12:33:33.170682 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:33.170687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:33.171702 (+ 1015us) server_negotiation.cc:407] Connection header received
0811 12:33:33.173696 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:33.173703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:33.173805 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:33.173901 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:33.175470 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:33.176256 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:33.177883 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:33.178342 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:33.178471 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:33.182690 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:33.182698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:33.182703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:33.182744 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:33.185936 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:33.186910 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:33.186915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:33.186918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:33.187088 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:33.187610 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:33.187615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:33.187618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:33.187989 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:33.188169 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:33.188949 (+ 780us) server_negotiation.cc:299] Negotiation successful
0811 12:33:33.189195 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
I20260811 12:33:33.190538 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:33.170797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37535 (local address 127.3.204.66:45391)
0811 12:33:33.171545 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:33.171583 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:33.173416 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:33.174153 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:33.174162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:33.174706 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:33.175300 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:33.175318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:33.176398 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:33.176402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:33.177738 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:33.177751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:33.178610 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:33.178618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:33.178905 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:33.179675 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:33.179699 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:33.182408 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:33.186145 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:33.186157 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:33.186176 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:33.186769 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:33.187193 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:33.187197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:33.187200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:33.187488 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:33.188140 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:33.188146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:33.188534 (+ 388us) client_negotiation.cc:769] Sending connection context
0811 12:33:33.188824 (+ 290us) client_negotiation.cc:240] Negotiation successful
0811 12:33:33.189271 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":279,"threads_started":1}
I20260811 12:33:33.192023 8945 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37535
I20260811 12:33:33.192399 8945 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:33.193236 8945 heartbeater.cc:511] Master 127.3.204.126:37535 requested a full tablet report, sending...
I20260811 12:33:33.194859 8346 ts_manager.cc:194] Registered new tserver with Master: 93af4564829a442499a0f2cf9f087e1b (127.3.204.66:34239)
I20260811 12:33:33.195703 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:45391
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:33.226682 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37535
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:33.561117 8954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:33.561429 8954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:33.561524 8954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:33.568925 8954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:33.569056 8954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:33.569169 8954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:33.579175 8954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:33.580632 8954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:33.581924 8954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:33.595031 8965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:33.595141 8971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:33.595149 8967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:33.595817 8954 server_base.cc:1034] running on GCE node
I20260811 12:33:33.596431 8954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:33.597388 8954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:33.598613 8954 hybrid_clock.cc:648] HybridClock initialized: now 1786451613598564 us; error 36 us; skew 500 ppm
I20260811 12:33:33.598917 8954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:33 dist-test-slave-lgr5 krb5kdc[8270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451613, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:33.602078 8954 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:33.603643 8954 webserver.cc:460] Webserver started at http://127.3.204.67:38211/ using document root <none> and password file <none>
I20260811 12:33:33.604423 8954 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:33.604539 8954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:33.604861 8954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:33.608029 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "93d3c7fa0f5e45daaa428ba3a5e19c06"
format_stamp: "Formatted at 2026-08-11 12:33:33 on dist-test-slave-lgr5"
server_key: "082723baeaa3c583edc69b2ea36d22cf"
server_key_iv: "64d796c7e92ab23bdd871639283fc20c"
server_key_version: "encryptionkey@0"
I20260811 12:33:33.608912 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "93d3c7fa0f5e45daaa428ba3a5e19c06"
format_stamp: "Formatted at 2026-08-11 12:33:33 on dist-test-slave-lgr5"
server_key: "082723baeaa3c583edc69b2ea36d22cf"
server_key_iv: "64d796c7e92ab23bdd871639283fc20c"
server_key_version: "encryptionkey@0"
I20260811 12:33:33.614117 8954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:33:33.617774 8986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:33.619139 8954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:33:33.619302 8954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "93d3c7fa0f5e45daaa428ba3a5e19c06"
format_stamp: "Formatted at 2026-08-11 12:33:33 on dist-test-slave-lgr5"
server_key: "082723baeaa3c583edc69b2ea36d22cf"
server_key_iv: "64d796c7e92ab23bdd871639283fc20c"
server_key_version: "encryptionkey@0"
I20260811 12:33:33.619491 8954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:33.645023 8954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:33.649178 8954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:33.649462 8954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:33.650419 8954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:33.651757 8954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:33.651832 8954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:33.651948 8954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:33.652006 8954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:33.668133 8697 heartbeater.cc:503] Master 127.3.204.126:37535 was elected leader, sending a full tablet report...
I20260811 12:33:33.683882 8954 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:41277
I20260811 12:33:33.683965 9177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:41277 every 8 connection(s)
I20260811 12:33:33.685262 8954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:33.687918 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8954
I20260811 12:33:33.688135 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:33.688575 3889 external_mini_cluster.cc:1428] Setting key 220d0990c089efa9c7ecb104894708e5
Aug 11 12:33:33 dist-test-slave-lgr5 krb5kdc[8270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451613, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:33.705586 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:33.687909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:45787 (local address 127.3.204.126:37535)
0811 12:33:33.688161 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:33.688167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:33.689413 (+ 1246us) server_negotiation.cc:407] Connection header received
0811 12:33:33.691721 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:33.691726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:33.691825 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:33.691911 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:33.693516 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:33.694268 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:33.695843 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:33.696283 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:33.696393 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:33.699742 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:33.699747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:33.699751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:33.699784 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:33.702521 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:33.703374 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:33.703378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:33.703381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:33.703564 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:33.703980 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:33.703984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:33.703986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:33.704360 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:33.704486 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:33.705159 (+ 673us) server_negotiation.cc:299] Negotiation successful
0811 12:33:33.705334 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 12:33:33.706614 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:33.688413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37535 (local address 127.3.204.67:45787)
0811 12:33:33.689249 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:33.689306 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:33.691412 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:33.692199 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:33.692208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:33.692743 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:33.693334 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:33.693353 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:33.694419 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:33.694433 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:33.695693 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:33.695707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:33.696529 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:33.696537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:33.696757 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:33.697363 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:33.697386 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:33.699545 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:33.702670 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:33.702679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:33.702696 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:33.703267 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:33.703668 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:33.703673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:33.703676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:33.703891 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:33.704501 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:33.704507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:33.704837 (+ 330us) client_negotiation.cc:769] Sending connection context
0811 12:33:33.705086 (+ 249us) client_negotiation.cc:240] Negotiation successful
0811 12:33:33.705519 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":308,"threads_started":1}
I20260811 12:33:33.707947 9178 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37535
I20260811 12:33:33.708267 9178 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:33.708987 9178 heartbeater.cc:511] Master 127.3.204.126:37535 requested a full tablet report, sending...
I20260811 12:33:33.710433 8346 ts_manager.cc:194] Registered new tserver with Master: 93d3c7fa0f5e45daaa428ba3a5e19c06 (127.3.204.67:41277)
I20260811 12:33:33.711232 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:45787
I20260811 12:33:33.716536 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 12:33:33.750986 8947 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:33:33.748578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.3.204.126:37535)
0811 12:33:33.748753 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:33.748758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:33.749214 (+ 456us) server_negotiation.cc:407] Connection header received
0811 12:33:33.749368 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:33.749372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:33.749489 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:33.749585 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:33.750774 (+ 1189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48140: BlockingRecv error: recv got EOF from 127.0.0.1:48140 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:33:33.754076 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8457
W20260811 12:33:33.756868 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8457: Not authorized: Client connection negotiation failed: client connection to 127.3.204.65:40767: server requires authentication, but client does not have Kerberos credentials available
I20260811 12:33:33.757036 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8457
I20260811 12:33:33.790613 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8707
W20260811 12:33:33.793301 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8707: Not authorized: Client connection negotiation failed: client connection to 127.3.204.66:34239: server requires authentication, but client does not have Kerberos credentials available
I20260811 12:33:33.793493 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8707
I20260811 12:33:33.816918 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8954
W20260811 12:33:33.819900 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8954: Not authorized: Client connection negotiation failed: client connection to 127.3.204.67:41277: server requires authentication, but client does not have Kerberos credentials available
I20260811 12:33:33.820101 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8954
I20260811 12:33:33.847767 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8289
I20260811 12:33:33.849100 8400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:33.952248 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8289
2026-08-11T12:33:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3816 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:34 dist-test-slave-lgr5 krb5kdc[9207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:34 dist-test-slave-lgr5 krb5kdc[9207](info): set up 2 sockets
Aug 11 12:33:34 dist-test-slave-lgr5 krb5kdc[9207](info): commencing operation
krb5kdc: starting...
W20260811 12:33:35.198791 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:35 dist-test-slave-lgr5 krb5kdc[9207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:35.415905 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:36869
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:36869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:35.747857 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:35.748219 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:35.748375 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:35.755697 9232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:35.755806 9232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:35.755880 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:35.755962 9232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:35.756026 9232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:35.766108 9232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:36869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:36869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:35.767455 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:35.768846 9232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:35.781051 9240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:35.780995 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:35.781025 9239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:35.781997 9232 server_base.cc:1034] running on GCE node
I20260811 12:33:35.783221 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:35.785075 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:35.786294 9232 hybrid_clock.cc:648] HybridClock initialized: now 1786451615786214 us; error 84 us; skew 500 ppm
I20260811 12:33:35.786689 9232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:35 dist-test-slave-lgr5 krb5kdc[9207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:35.794973 9232 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:35.796720 9232 webserver.cc:460] Webserver started at http://127.3.204.126:36509/ using document root <none> and password file <none>
I20260811 12:33:35.797458 9232 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:35.797580 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:35.797909 9232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:35.801028 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "0a0f32394d9d428089684ece8d17bf83"
format_stamp: "Formatted at 2026-08-11 12:33:35 on dist-test-slave-lgr5"
server_key: "96e2da2e471e2a04c91ef0e420a7b0b3"
server_key_iv: "7563847e1549ccfeeca87275f8904ed0"
server_key_version: "encryptionkey@0"
I20260811 12:33:35.801806 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "0a0f32394d9d428089684ece8d17bf83"
format_stamp: "Formatted at 2026-08-11 12:33:35 on dist-test-slave-lgr5"
server_key: "96e2da2e471e2a04c91ef0e420a7b0b3"
server_key_iv: "7563847e1549ccfeeca87275f8904ed0"
server_key_version: "encryptionkey@0"
I20260811 12:33:35.807238 9232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 12:33:35.810768 9262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:35.812191 9232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 12:33:35.812364 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "0a0f32394d9d428089684ece8d17bf83"
format_stamp: "Formatted at 2026-08-11 12:33:35 on dist-test-slave-lgr5"
server_key: "96e2da2e471e2a04c91ef0e420a7b0b3"
server_key_iv: "7563847e1549ccfeeca87275f8904ed0"
server_key_version: "encryptionkey@0"
I20260811 12:33:35.812561 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:35.828361 9232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:35.833001 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:35.833272 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:35.851158 9232 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:36869
I20260811 12:33:35.851207 9350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:36869 every 8 connection(s)
I20260811 12:33:35.852610 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:35.856324 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:35.857062 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9232
I20260811 12:33:35.857280 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:35.858120 3889 external_mini_cluster.cc:1428] Setting key bcc8f0046d34002ee334dace0a8d9a99
I20260811 12:33:35.871065 9351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83: Bootstrap starting.
Aug 11 12:33:35 dist-test-slave-lgr5 krb5kdc[9207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:35.875145 9351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:35.876397 9351 log.cc:824] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:35.879449 9351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83: No bootstrap required, opened a new log
I20260811 12:33:35.881700 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:35.860103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.3.204.126:36869)
0811 12:33:35.860889 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:35.860901 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:35.860942 (+ 41us) server_negotiation.cc:407] Connection header received
0811 12:33:35.862712 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:35.862736 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:35.863180 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:35.863526 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:35.865576 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:35.866872 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:35.868723 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:35.869405 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:35.869541 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:35.874303 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:35.874316 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:35.874336 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:35.874389 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:35.877603 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:35.878303 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:35.878309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:35.878316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:35.878496 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:35.878855 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:35.878859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:35.878861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:35.879409 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:35.879708 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:35.880215 (+ 507us) server_negotiation.cc:299] Negotiation successful
0811 12:33:35.880642 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":336,"threads_started":1}
I20260811 12:33:35.885725 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:33:35.888063 9351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0f32394d9d428089684ece8d17bf83" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36869 } }
I20260811 12:33:35.888531 9351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:35.888666 9351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0f32394d9d428089684ece8d17bf83, State: Initialized, Role: FOLLOWER
I20260811 12:33:35.889400 9351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0f32394d9d428089684ece8d17bf83" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36869 } }
I20260811 12:33:35.889643 9351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:35.889798 9351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:35.889982 9351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:35.892397 9351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0f32394d9d428089684ece8d17bf83" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36869 } }
I20260811 12:33:35.892972 9351 leader_election.cc:302] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a0f32394d9d428089684ece8d17bf83; no voters:
I20260811 12:33:35.893666 9351 leader_election.cc:288] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:35.893779 9359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:35.895332 9359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [term 1 LEADER]: Becoming Leader. State: Replica: 0a0f32394d9d428089684ece8d17bf83, State: Running, Role: LEADER
I20260811 12:33:35.896206 9359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0f32394d9d428089684ece8d17bf83" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36869 } }
I20260811 12:33:35.897835 9351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:35.899297 9362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0f32394d9d428089684ece8d17bf83. Latest consensus state: current_term: 1 leader_uuid: "0a0f32394d9d428089684ece8d17bf83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0f32394d9d428089684ece8d17bf83" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36869 } } }
I20260811 12:33:35.899667 9362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:35.899391 9363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a0f32394d9d428089684ece8d17bf83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0f32394d9d428089684ece8d17bf83" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36869 } } }
I20260811 12:33:35.899788 9363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:35.900595 9369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:35.913142 9369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:35.936677 9369 catalog_manager.cc:1347] Generated new cluster ID: aae3bd24fbc44ff0873d100215308953
I20260811 12:33:35.936825 9369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:35.958779 9369 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:35.960947 9369 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:35.971048 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:35.953875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.3.204.126:36869)
0811 12:33:35.954080 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:35.954086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:35.961229 (+ 7143us) server_negotiation.cc:407] Connection header received
0811 12:33:35.961769 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:35.961774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:35.961867 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:35.961958 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:35.963384 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:35.964522 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:35.966363 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:35.966995 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:35.967076 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:35.968604 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:35.968610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:35.968615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:35.969668 (+ 1053us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:35.969754 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:35.969790 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:35.969899 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:35.970595 (+ 696us) server_negotiation.cc:299] Negotiation successful
0811 12:33:35.970692 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 12:33:35.971940 9369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a0f32394d9d428089684ece8d17bf83: Generated new TSK 0
I20260811 12:33:35.973101 9369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:33:35.989885 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9232
I20260811 12:33:35.991815 9339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:36.082093 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9232
2026-08-11T12:33:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2123 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:36 dist-test-slave-lgr5 krb5kdc[9411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:36 dist-test-slave-lgr5 krb5kdc[9411](info): set up 2 sockets
Aug 11 12:33:36 dist-test-slave-lgr5 krb5kdc[9411](info): commencing operation
krb5kdc: starting...
W20260811 12:33:37.325006 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:37 dist-test-slave-lgr5 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:37.545104 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:38957
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:37.876754 9428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:37.877096 9428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:37.877246 9428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:37.884640 9428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:37.884771 9428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:37.884852 9428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:37.884929 9428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:37.884990 9428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:37.895220 9428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:37.896680 9428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:37.898023 9428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:37.909725 9433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:37.909919 9434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:37.909885 9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:37.910821 9428 server_base.cc:1034] running on GCE node
I20260811 12:33:37.911654 9428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:37.913167 9428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:37.914367 9428 hybrid_clock.cc:648] HybridClock initialized: now 1786451617914309 us; error 51 us; skew 500 ppm
I20260811 12:33:37.914743 9428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:37 dist-test-slave-lgr5 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:37.917987 9428 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:37.919515 9428 webserver.cc:460] Webserver started at http://127.3.204.126:34273/ using document root <none> and password file <none>
I20260811 12:33:37.920265 9428 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:37.920377 9428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:37.920665 9428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:37.923761 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "b3170e79c1614ce183dc8f57b9c4c4fe"
format_stamp: "Formatted at 2026-08-11 12:33:37 on dist-test-slave-lgr5"
server_key: "6c898436f833833b027b4262108a514b"
server_key_iv: "1bea2d51d96b8f054de6f0ad1eb2306b"
server_key_version: "encryptionkey@0"
I20260811 12:33:37.924661 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "b3170e79c1614ce183dc8f57b9c4c4fe"
format_stamp: "Formatted at 2026-08-11 12:33:37 on dist-test-slave-lgr5"
server_key: "6c898436f833833b027b4262108a514b"
server_key_iv: "1bea2d51d96b8f054de6f0ad1eb2306b"
server_key_version: "encryptionkey@0"
I20260811 12:33:37.930207 9428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:33:37.933816 9455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:37.935297 9428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:33:37.935460 9428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "b3170e79c1614ce183dc8f57b9c4c4fe"
format_stamp: "Formatted at 2026-08-11 12:33:37 on dist-test-slave-lgr5"
server_key: "6c898436f833833b027b4262108a514b"
server_key_iv: "1bea2d51d96b8f054de6f0ad1eb2306b"
server_key_version: "encryptionkey@0"
I20260811 12:33:37.935654 9428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:37.955184 9428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:37.959563 9428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:37.959851 9428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:37.980373 9428 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:38957
I20260811 12:33:37.980424 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:38957 every 8 connection(s)
I20260811 12:33:37.982074 9428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:37.986583 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9428
I20260811 12:33:37.986820 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:37.987663 3889 external_mini_cluster.cc:1428] Setting key 46a3ae1cd219a911285168483aa07b61
I20260811 12:33:37.987932 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:38.006297 9538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe: Bootstrap starting.
Aug 11 12:33:38 dist-test-slave-lgr5 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:38.010069 9538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:38.011426 9538 log.cc:824] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:38.015398 9538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe: No bootstrap required, opened a new log
I20260811 12:33:38.019969 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:37.990443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.3.204.126:38957)
0811 12:33:37.991367 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:37.991389 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:37.991458 (+ 69us) server_negotiation.cc:407] Connection header received
0811 12:33:37.994048 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:37.994081 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:37.994670 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:37.995168 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:37.997028 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:37.998865 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:38.000679 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:38.001862 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:38.002072 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:38.010784 (+ 8712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:38.010796 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:38.010818 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:38.010878 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:38.014900 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:38.015745 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:38.015754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:38.015764 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:38.016009 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:38.016475 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:38.016481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:38.016484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:38.017239 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:38.017577 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:38.017951 (+ 374us) server_negotiation.cc:299] Negotiation successful
0811 12:33:38.018479 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":352,"threads_started":1}
I20260811 12:33:38.024669 9538 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3170e79c1614ce183dc8f57b9c4c4fe" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38957 } }
I20260811 12:33:38.025060 9538 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:38.025141 9538 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3170e79c1614ce183dc8f57b9c4c4fe, State: Initialized, Role: FOLLOWER
I20260811 12:33:38.025658 9538 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3170e79c1614ce183dc8f57b9c4c4fe" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38957 } }
I20260811 12:33:38.025867 9538 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:38.025970 9538 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:38.026095 9538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:38.028373 9538 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3170e79c1614ce183dc8f57b9c4c4fe" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38957 } }
I20260811 12:33:38.028895 9538 leader_election.cc:302] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3170e79c1614ce183dc8f57b9c4c4fe; no voters:
I20260811 12:33:38.029525 9538 leader_election.cc:288] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:38.029640 9548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:38.030287 9548 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [term 1 LEADER]: Becoming Leader. State: Replica: b3170e79c1614ce183dc8f57b9c4c4fe, State: Running, Role: LEADER
I20260811 12:33:38.030911 9548 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3170e79c1614ce183dc8f57b9c4c4fe" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38957 } }
I20260811 12:33:38.031523 9538 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:38.033769 9550 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3170e79c1614ce183dc8f57b9c4c4fe. Latest consensus state: current_term: 1 leader_uuid: "b3170e79c1614ce183dc8f57b9c4c4fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3170e79c1614ce183dc8f57b9c4c4fe" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38957 } } }
I20260811 12:33:38.034087 9550 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:38.034714 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3170e79c1614ce183dc8f57b9c4c4fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3170e79c1614ce183dc8f57b9c4c4fe" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38957 } } }
I20260811 12:33:38.034941 9557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:38.035085 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:38.041039 9557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:38.051389 9557 catalog_manager.cc:1347] Generated new cluster ID: 7bfe8a41f81a42e09f3e92f2b320e7ad
I20260811 12:33:38.051522 9557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:38.075664 9557 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:38.077092 9557 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:38.088698 9557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3170e79c1614ce183dc8f57b9c4c4fe: Generated new TSK 0
I20260811 12:33:38.089619 9557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:38.143723 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38957
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:38.479635 9578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:38.479941 9578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:38.480041 9578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:38.487352 9578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:38.487480 9578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:38.487596 9578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:38.498138 9578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:38.499509 9578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:38.500873 9578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:38.514451 9588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:38.514374 9591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:38.514390 9589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:38.515851 9578 server_base.cc:1034] running on GCE node
I20260811 12:33:38.516597 9578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:38.517616 9578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:38.518862 9578 hybrid_clock.cc:648] HybridClock initialized: now 1786451618518798 us; error 45 us; skew 500 ppm
I20260811 12:33:38.519261 9578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:38 dist-test-slave-lgr5 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:38.522683 9578 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:38.524258 9578 webserver.cc:460] Webserver started at http://127.3.204.65:43975/ using document root <none> and password file <none>
I20260811 12:33:38.524935 9578 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:38.525043 9578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:38.525326 9578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:38.528519 9578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "7c9eb84bc9a94703be052aa258cdca01"
format_stamp: "Formatted at 2026-08-11 12:33:38 on dist-test-slave-lgr5"
server_key: "87383daf9e0c6a4a981b02b571bfe0c2"
server_key_iv: "fed89381676b5a78288f82111c5f6867"
server_key_version: "encryptionkey@0"
I20260811 12:33:38.529289 9578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "7c9eb84bc9a94703be052aa258cdca01"
format_stamp: "Formatted at 2026-08-11 12:33:38 on dist-test-slave-lgr5"
server_key: "87383daf9e0c6a4a981b02b571bfe0c2"
server_key_iv: "fed89381676b5a78288f82111c5f6867"
server_key_version: "encryptionkey@0"
I20260811 12:33:38.534550 9578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260811 12:33:38.538424 9610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:38.539944 9578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:33:38.540124 9578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "7c9eb84bc9a94703be052aa258cdca01"
format_stamp: "Formatted at 2026-08-11 12:33:38 on dist-test-slave-lgr5"
server_key: "87383daf9e0c6a4a981b02b571bfe0c2"
server_key_iv: "fed89381676b5a78288f82111c5f6867"
server_key_version: "encryptionkey@0"
I20260811 12:33:38.540349 9578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:38.557737 9578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:38.562239 9578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:38.562534 9578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:38.563560 9578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:38.565061 9578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:38.565135 9578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:38.565258 9578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:38.565312 9578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:38.596519 9578 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:34677
I20260811 12:33:38.596562 9802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:34677 every 8 connection(s)
I20260811 12:33:38.597867 9578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:38.606366 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9578
I20260811 12:33:38.606588 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:38.607304 3889 external_mini_cluster.cc:1428] Setting key ad121785b4264060b231289f5b95cae8
Aug 11 12:33:38 dist-test-slave-lgr5 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:38.618643 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:38.600443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:46621 (local address 127.3.204.126:38957)
0811 12:33:38.600900 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:38.600905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:38.601905 (+ 1000us) server_negotiation.cc:407] Connection header received
0811 12:33:38.603727 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:38.603732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:38.603815 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:38.603914 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:38.605561 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:38.606299 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:38.608002 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:38.608521 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:38.608651 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:38.612774 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:38.612780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:38.612783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:38.612821 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:38.615323 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:38.616153 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:38.616158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:38.616161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:38.616295 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:38.616760 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:38.616765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:38.616767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:38.617075 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:38.617219 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:38.618011 (+ 792us) server_negotiation.cc:299] Negotiation successful
0811 12:33:38.618265 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260811 12:33:38.619656 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:38.601018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:38957 (local address 127.3.204.65:46621)
0811 12:33:38.601726 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:38.601790 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:38.603440 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:38.604205 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:38.604213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:38.604766 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:38.605356 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:38.605374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:38.606462 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:38.606468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:38.607842 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:38.607856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:38.608778 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:38.608786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:38.609036 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:38.609763 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:38.609794 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:38.612520 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:38.615484 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:38.615494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:38.615515 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:38.615995 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:38.616412 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:38.616418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:38.616422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:38.616640 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:38.617213 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:38.617222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:38.617614 (+ 392us) client_negotiation.cc:769] Sending connection context
0811 12:33:38.617886 (+ 272us) client_negotiation.cc:240] Negotiation successful
0811 12:33:38.618484 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":259,"threads_started":1}
I20260811 12:33:38.621338 9805 heartbeater.cc:347] Connected to a master server at 127.3.204.126:38957
I20260811 12:33:38.621688 9805 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:38.622643 9805 heartbeater.cc:511] Master 127.3.204.126:38957 requested a full tablet report, sending...
I20260811 12:33:38.625231 9484 ts_manager.cc:194] Registered new tserver with Master: 7c9eb84bc9a94703be052aa258cdca01 (127.3.204.65:34677)
I20260811 12:33:38.627300 9484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:46621
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:38.660390 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38957
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:39.003655 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:39.003979 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:39.004165 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:39.011709 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:39.011847 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:39.012009 9817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:39.022132 9817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:39.023562 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:39.024978 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:39.038636 9826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:39.038618 9825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:39.039050 9831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:39.039698 9817 server_base.cc:1034] running on GCE node
I20260811 12:33:39.040441 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:39.041501 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:39.042747 9817 hybrid_clock.cc:648] HybridClock initialized: now 1786451619042691 us; error 45 us; skew 500 ppm
I20260811 12:33:39.043119 9817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:39.046622 9817 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:39.048156 9817 webserver.cc:460] Webserver started at http://127.3.204.66:35623/ using document root <none> and password file <none>
I20260811 12:33:39.048863 9817 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:39.048985 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:39.049275 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:39.052275 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "eb8b9e58673e4a78a689d4a41177bb85"
format_stamp: "Formatted at 2026-08-11 12:33:39 on dist-test-slave-lgr5"
server_key: "e78a10c54259a8e1aef40405d65edcb2"
server_key_iv: "f1022355535ef1f21878d66db331b19c"
server_key_version: "encryptionkey@0"
I20260811 12:33:39.053067 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "eb8b9e58673e4a78a689d4a41177bb85"
format_stamp: "Formatted at 2026-08-11 12:33:39 on dist-test-slave-lgr5"
server_key: "e78a10c54259a8e1aef40405d65edcb2"
server_key_iv: "f1022355535ef1f21878d66db331b19c"
server_key_version: "encryptionkey@0"
I20260811 12:33:39.058506 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:33:39.062389 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:39.063922 9817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:33:39.064145 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "eb8b9e58673e4a78a689d4a41177bb85"
format_stamp: "Formatted at 2026-08-11 12:33:39 on dist-test-slave-lgr5"
server_key: "e78a10c54259a8e1aef40405d65edcb2"
server_key_iv: "f1022355535ef1f21878d66db331b19c"
server_key_version: "encryptionkey@0"
I20260811 12:33:39.064364 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:39.085875 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:39.090548 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:39.090849 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:39.091843 9817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:39.093302 9817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:39.093389 9817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:39.093514 9817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:39.093568 9817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:39.124861 9817 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:39289
I20260811 12:33:39.124948 10036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:39289 every 8 connection(s)
I20260811 12:33:39.126260 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:39.132557 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9817
I20260811 12:33:39.132757 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:39.133368 3889 external_mini_cluster.cc:1428] Setting key cda03aef687382cb84de2e2ffc74f698
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:39.149281 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:39.128929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:49913 (local address 127.3.204.126:38957)
0811 12:33:39.129405 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:39.129409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:39.130300 (+ 891us) server_negotiation.cc:407] Connection header received
0811 12:33:39.132290 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:39.132295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:39.132369 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:39.132489 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:39.134575 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.135341 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.137741 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.138194 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.138295 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:39.142669 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:39.142675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:39.142678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:39.142718 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:39.145860 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.146709 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.146714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.146717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.146871 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.147356 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.147360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.147362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.147664 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:39.147806 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:39.148649 (+ 843us) server_negotiation.cc:299] Negotiation successful
0811 12:33:39.148910 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":239,"threads_started":1}
I20260811 12:33:39.150341 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:39.129369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:38957 (local address 127.3.204.66:49913)
0811 12:33:39.130118 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:39.130171 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:39.131957 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:39.132776 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:39.132789 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:39.133604 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:39.134378 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:39.134400 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.135509 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.135514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:39.137556 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:39.137573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.138433 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.138440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:39.138818 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:39.139589 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:39.139645 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:39.142402 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:39.146048 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:39.146057 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:39.146073 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:39.146565 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:39.147005 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:39.147009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:39.147011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:39.147221 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:39.147804 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:39.147814 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:39.148242 (+ 428us) client_negotiation.cc:769] Sending connection context
0811 12:33:39.148506 (+ 264us) client_negotiation.cc:240] Negotiation successful
0811 12:33:39.149127 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":282,"threads_started":1}
I20260811 12:33:39.151901 10037 heartbeater.cc:347] Connected to a master server at 127.3.204.126:38957
I20260811 12:33:39.152271 10037 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:39.153049 10037 heartbeater.cc:511] Master 127.3.204.126:38957 requested a full tablet report, sending...
I20260811 12:33:39.154820 9484 ts_manager.cc:194] Registered new tserver with Master: eb8b9e58673e4a78a689d4a41177bb85 (127.3.204.66:39289)
I20260811 12:33:39.155704 9484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:49913
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:39.187187 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38957
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:39.516882 10045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:39.517194 10045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:39.517302 10045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:39.524721 10045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:39.524854 10045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:39.524976 10045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:39.534857 10045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:39.536237 10045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:39.537599 10045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:39.550971 10053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:39.550937 10054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:39.551157 10056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:39.551723 10045 server_base.cc:1034] running on GCE node
I20260811 12:33:39.552440 10045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:39.553457 10045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:39.554687 10045 hybrid_clock.cc:648] HybridClock initialized: now 1786451619554618 us; error 54 us; skew 500 ppm
I20260811 12:33:39.555061 10045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:39.558858 10045 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:39.560563 10045 webserver.cc:460] Webserver started at http://127.3.204.67:35615/ using document root <none> and password file <none>
I20260811 12:33:39.561278 10045 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:39.561396 10045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:39.561702 10045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:39.564802 10045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "e5046a1abbaf4953ace08ff8462b29a5"
format_stamp: "Formatted at 2026-08-11 12:33:39 on dist-test-slave-lgr5"
server_key: "0f83170d4af2e5474133b9c18229ed17"
server_key_iv: "6fdb2a15ba011ec6f6436980e07d6505"
server_key_version: "encryptionkey@0"
I20260811 12:33:39.565598 10045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "e5046a1abbaf4953ace08ff8462b29a5"
format_stamp: "Formatted at 2026-08-11 12:33:39 on dist-test-slave-lgr5"
server_key: "0f83170d4af2e5474133b9c18229ed17"
server_key_iv: "6fdb2a15ba011ec6f6436980e07d6505"
server_key_version: "encryptionkey@0"
I20260811 12:33:39.570900 10045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:39.574457 10075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:39.575814 10045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 12:33:39.575985 10045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "e5046a1abbaf4953ace08ff8462b29a5"
format_stamp: "Formatted at 2026-08-11 12:33:39 on dist-test-slave-lgr5"
server_key: "0f83170d4af2e5474133b9c18229ed17"
server_key_iv: "6fdb2a15ba011ec6f6436980e07d6505"
server_key_version: "encryptionkey@0"
I20260811 12:33:39.576203 10045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:39.600020 10045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:39.604583 10045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:39.604885 10045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:39.605911 10045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:39.607264 10045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:39.607340 10045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:39.607462 10045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:39.607514 10045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:39.630378 9805 heartbeater.cc:503] Master 127.3.204.126:38957 was elected leader, sending a full tablet report...
I20260811 12:33:39.639631 10045 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:40815
I20260811 12:33:39.639708 10260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:40815 every 8 connection(s)
I20260811 12:33:39.641078 10045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:39.648721 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10045
I20260811 12:33:39.648897 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:39.649359 3889 external_mini_cluster.cc:1428] Setting key 25a93d2760d8cf6d6b1993eba803c73d
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:39.661723 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:39.644309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44789 (local address 127.3.204.126:38957)
0811 12:33:39.644547 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:39.644554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:39.645706 (+ 1152us) server_negotiation.cc:407] Connection header received
0811 12:33:39.647573 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:39.647578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:39.647675 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:39.647768 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:39.649535 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.650494 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.652259 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.652652 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.652764 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:39.656235 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:39.656240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:39.656244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:39.656286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:39.658687 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.659411 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.659415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.659418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.659542 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.659966 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.659970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.659972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.660298 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:39.660440 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:39.661160 (+ 720us) server_negotiation.cc:299] Negotiation successful
0811 12:33:39.661402 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:33:39.662668 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:39.644716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:38957 (local address 127.3.204.67:44789)
0811 12:33:39.645539 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:39.645578 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:39.647277 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:39.647985 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:39.647993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:39.648672 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:39.649331 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:39.649351 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.650651 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.650655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:39.652067 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:39.652126 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.652891 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.652900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:39.653119 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:39.653760 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:39.653798 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:39.656004 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:39.658829 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:39.658839 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:39.658858 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:39.659286 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:39.659644 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:39.659649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:39.659653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:39.659852 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:39.660421 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:39.660429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:39.660772 (+ 343us) client_negotiation.cc:769] Sending connection context
0811 12:33:39.661025 (+ 253us) client_negotiation.cc:240] Negotiation successful
0811 12:33:39.661576 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":311,"threads_started":1}
I20260811 12:33:39.664125 10261 heartbeater.cc:347] Connected to a master server at 127.3.204.126:38957
I20260811 12:33:39.664423 10261 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:39.665179 10261 heartbeater.cc:511] Master 127.3.204.126:38957 requested a full tablet report, sending...
I20260811 12:33:39.666636 9483 ts_manager.cc:194] Registered new tserver with Master: e5046a1abbaf4953ace08ff8462b29a5 (127.3.204.67:40815)
I20260811 12:33:39.667506 9483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:44789
I20260811 12:33:39.677122 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:39.729504 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:39.715716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.3.204.126:38957)
0811 12:33:39.715951 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:39.715965 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:39.716352 (+ 387us) server_negotiation.cc:407] Connection header received
0811 12:33:39.716475 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:39.716479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:39.716570 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:39.716662 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:39.718046 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.718807 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.720159 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.720585 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.720679 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:39.724459 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:39.724465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:39.724468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:39.724509 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:39.727100 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.727731 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.727735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.727738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.727859 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.728222 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.728225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.728227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.728561 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:39.728767 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:39.728992 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 12:33:39.729169 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260811 12:33:39.730129 9483 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49100
I20260811 12:33:39.731638 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9578
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:39.747980 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:39.732854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.3.204.65:34677)
0811 12:33:39.733468 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:39.733475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:39.733501 (+ 26us) server_negotiation.cc:407] Connection header received
0811 12:33:39.733561 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:39.733567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:39.733805 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:39.734001 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:39.735297 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.736412 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.737661 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.738313 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.738423 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:39.742488 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:39.742503 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:39.742520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:39.742560 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:39.745477 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.746140 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.746148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.746150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.746290 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.746589 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.746592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.746594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.746959 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:39.747188 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:39.747388 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 12:33:39.747591 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":292,"threads_started":1}
W20260811 12:33:39.749270 9793 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50080
W20260811 12:33:39.750414 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:39.750630 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9578
I20260811 12:33:39.777300 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9817
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:33:39.793740 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:39.778233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.3.204.66:39289)
0811 12:33:39.778764 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:39.778772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:39.778799 (+ 27us) server_negotiation.cc:407] Connection header received
0811 12:33:39.778876 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:39.778883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:39.779104 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:39.779267 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:39.780576 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.781706 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.782995 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.783629 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.783741 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:39.787882 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:39.787898 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:39.787915 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:39.787958 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:39.791007 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.791717 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.791726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.791729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.791866 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.792211 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.792215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.792218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.792652 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:39.792904 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:39.793182 (+ 278us) server_negotiation.cc:299] Negotiation successful
0811 12:33:39.793426 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":253,"threads_started":1}
W20260811 12:33:39.795231 10032 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39830
W20260811 12:33:39.796555 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9817: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:39.796723 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9817
I20260811 12:33:39.827839 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10045
Aug 11 12:33:39 dist-test-slave-lgr5 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451619, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:33:39.848683 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:39.829049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.3.204.67:40815)
0811 12:33:39.829791 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:39.829801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:39.829839 (+ 38us) server_negotiation.cc:407] Connection header received
0811 12:33:39.829916 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:39.829925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:39.830211 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:39.830442 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:39.832241 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.833419 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.834842 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:39.835741 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:39.836462 (+ 721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:39.841491 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:39.841510 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:39.841533 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:39.841593 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:39.845247 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.846148 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.846157 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.846160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.846336 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:39.846811 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:39.846816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:39.846819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:39.847256 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:39.847581 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:39.847963 (+ 382us) server_negotiation.cc:299] Negotiation successful
0811 12:33:39.848291 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":353,"threads_started":1}
W20260811 12:33:39.850265 10252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42814
W20260811 12:33:39.851934 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10045: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:33:39.852164 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10045
I20260811 12:33:39.878263 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9428
I20260811 12:33:39.879604 9531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:39.985378 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9428
2026-08-11T12:33:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3903 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:40 dist-test-slave-lgr5 krb5kdc[10286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:40 dist-test-slave-lgr5 krb5kdc[10286](info): set up 2 sockets
Aug 11 12:33:40 dist-test-slave-lgr5 krb5kdc[10286](info): commencing operation
krb5kdc: starting...
W20260811 12:33:41.236240 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:41 dist-test-slave-lgr5 krb5kdc[10286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:41.464293 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:36079
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:36079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:41.797614 10313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:41.797926 10313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:41.798038 10313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:41.806418 10313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:41.806542 10313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:41.806615 10313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:41.806692 10313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:41.806751 10313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:41.816999 10313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:36079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:36079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:41.818382 10313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:41.819717 10313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:41.831602 10321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:41.831686 10320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:41.831701 10323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:41.832515 10313 server_base.cc:1034] running on GCE node
I20260811 12:33:41.833355 10313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:41.834887 10313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:41.836146 10313 hybrid_clock.cc:648] HybridClock initialized: now 1786451621836059 us; error 36 us; skew 500 ppm
I20260811 12:33:41.836651 10313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:41 dist-test-slave-lgr5 krb5kdc[10286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451621, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:41.844909 10313 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:41.846587 10313 webserver.cc:460] Webserver started at http://127.3.204.126:33127/ using document root <none> and password file <none>
I20260811 12:33:41.847338 10313 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:41.847462 10313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:41.847783 10313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:41.850854 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "c9afd347e924411c98df91302b0dd1b9"
format_stamp: "Formatted at 2026-08-11 12:33:41 on dist-test-slave-lgr5"
server_key: "0e0fb97213e3cfb52e30030985dc9315"
server_key_iv: "0e061ff5a43b4b907a896287233b3545"
server_key_version: "encryptionkey@0"
I20260811 12:33:41.851642 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "c9afd347e924411c98df91302b0dd1b9"
format_stamp: "Formatted at 2026-08-11 12:33:41 on dist-test-slave-lgr5"
server_key: "0e0fb97213e3cfb52e30030985dc9315"
server_key_iv: "0e061ff5a43b4b907a896287233b3545"
server_key_version: "encryptionkey@0"
I20260811 12:33:41.856739 10313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 12:33:41.860492 10334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:41.861951 10313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:33:41.862202 10313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "c9afd347e924411c98df91302b0dd1b9"
format_stamp: "Formatted at 2026-08-11 12:33:41 on dist-test-slave-lgr5"
server_key: "0e0fb97213e3cfb52e30030985dc9315"
server_key_iv: "0e061ff5a43b4b907a896287233b3545"
server_key_version: "encryptionkey@0"
I20260811 12:33:41.862422 10313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:41.876648 10313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:41.880882 10313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:41.881121 10313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:41.899276 10313 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:36079
I20260811 12:33:41.899351 10405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:36079 every 8 connection(s)
I20260811 12:33:41.900770 10313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:41.904587 10407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:41.906232 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10313
I20260811 12:33:41.906471 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:41.907328 3889 external_mini_cluster.cc:1428] Setting key 2425935839c9e59f041a2923aff6b93f
I20260811 12:33:41.916889 10407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9: Bootstrap starting.
I20260811 12:33:41.919976 10407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:41.921088 10407 log.cc:824] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9: Log is configured to *not* fsync() on all Append() calls
Aug 11 12:33:41 dist-test-slave-lgr5 krb5kdc[10286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:41.924785 10407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9: No bootstrap required, opened a new log
I20260811 12:33:41.931396 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:41.909664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.3.204.126:36079)
0811 12:33:41.910482 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:41.910498 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:41.910544 (+ 46us) server_negotiation.cc:407] Connection header received
0811 12:33:41.912348 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:41.912373 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:41.912774 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:41.913146 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:41.914685 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:41.916052 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:41.917361 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:41.918054 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:41.918255 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:41.922561 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:41.922573 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:41.922594 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:41.922642 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:41.926560 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:41.927308 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:41.927316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:41.927333 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:41.927563 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:41.928021 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:41.928026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:41.928030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:41.928820 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:41.929121 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:41.929501 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 12:33:41.929971 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":332,"threads_started":1}
I20260811 12:33:41.934749 10407 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9afd347e924411c98df91302b0dd1b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36079 } }
I20260811 12:33:41.935112 10407 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:41.935191 10407 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9afd347e924411c98df91302b0dd1b9, State: Initialized, Role: FOLLOWER
I20260811 12:33:41.935684 10407 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9afd347e924411c98df91302b0dd1b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36079 } }
I20260811 12:33:41.935897 10407 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:41.935995 10407 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:41.936158 10407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:41.938609 10407 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9afd347e924411c98df91302b0dd1b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36079 } }
I20260811 12:33:41.939095 10407 leader_election.cc:302] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9afd347e924411c98df91302b0dd1b9; no voters:
I20260811 12:33:41.939666 10407 leader_election.cc:288] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:41.939795 10414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:41.940755 10414 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [term 1 LEADER]: Becoming Leader. State: Replica: c9afd347e924411c98df91302b0dd1b9, State: Running, Role: LEADER
I20260811 12:33:41.941561 10414 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9afd347e924411c98df91302b0dd1b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36079 } }
I20260811 12:33:41.941772 10407 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:41.944478 10419 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9afd347e924411c98df91302b0dd1b9. Latest consensus state: current_term: 1 leader_uuid: "c9afd347e924411c98df91302b0dd1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9afd347e924411c98df91302b0dd1b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36079 } } }
I20260811 12:33:41.944566 10415 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9afd347e924411c98df91302b0dd1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9afd347e924411c98df91302b0dd1b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36079 } } }
I20260811 12:33:41.944901 10419 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:41.944949 10415 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:41.947906 10428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:41.953323 10428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:41.962750 10428 catalog_manager.cc:1347] Generated new cluster ID: 8b21840d4726489c8114ab6ebe424247
I20260811 12:33:41.962880 10428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:41.980403 10428 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:41.982300 10428 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:41.997439 10428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9afd347e924411c98df91302b0dd1b9: Generated new TSK 0
I20260811 12:33:41.998581 10428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:42.064649 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36079
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:42.406198 10452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:42.406522 10452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:42.406657 10452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:42.413976 10452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:42.414124 10452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:42.414284 10452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:42.424226 10452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:42.425585 10452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:42.426927 10452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:42.440231 10460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:42.440258 10464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:42.440186 10462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:42.441017 10452 server_base.cc:1034] running on GCE node
I20260811 12:33:42.441646 10452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:42.442680 10452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:42.443907 10452 hybrid_clock.cc:648] HybridClock initialized: now 1786451622443833 us; error 60 us; skew 500 ppm
I20260811 12:33:42.444280 10452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:42 dist-test-slave-lgr5 krb5kdc[10286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:42.447561 10452 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:42.449198 10452 webserver.cc:460] Webserver started at http://127.3.204.65:44801/ using document root <none> and password file <none>
I20260811 12:33:42.449884 10452 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:42.450004 10452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:42.450341 10452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:42.453392 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "2142db33557b41e59083daa274a1f283"
format_stamp: "Formatted at 2026-08-11 12:33:42 on dist-test-slave-lgr5"
server_key: "a7a18de104002b7ed743fa4de55a6695"
server_key_iv: "3fa4e6ca5a5c7140a868c82a8b3c6e55"
server_key_version: "encryptionkey@0"
I20260811 12:33:42.454279 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "2142db33557b41e59083daa274a1f283"
format_stamp: "Formatted at 2026-08-11 12:33:42 on dist-test-slave-lgr5"
server_key: "a7a18de104002b7ed743fa4de55a6695"
server_key_iv: "3fa4e6ca5a5c7140a868c82a8b3c6e55"
server_key_version: "encryptionkey@0"
I20260811 12:33:42.459659 10452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:33:42.463687 10473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:42.465255 10452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:42.465461 10452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "2142db33557b41e59083daa274a1f283"
format_stamp: "Formatted at 2026-08-11 12:33:42 on dist-test-slave-lgr5"
server_key: "a7a18de104002b7ed743fa4de55a6695"
server_key_iv: "3fa4e6ca5a5c7140a868c82a8b3c6e55"
server_key_version: "encryptionkey@0"
I20260811 12:33:42.465674 10452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:42.477658 10452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:42.482034 10452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:42.482318 10452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:42.483322 10452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:42.484735 10452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:42.484822 10452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:42.484946 10452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:42.485000 10452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:42.516129 10452 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:43787
I20260811 12:33:42.516197 10660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:43787 every 8 connection(s)
I20260811 12:33:42.517459 10452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:42.518687 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10452
I20260811 12:33:42.518908 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:42.519706 3889 external_mini_cluster.cc:1428] Setting key 8d8ba7cb2e2a0154fd69d067cf704cbf
Aug 11 12:33:42 dist-test-slave-lgr5 krb5kdc[10286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:42.538525 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:42.520262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:60077 (local address 127.3.204.126:36079)
0811 12:33:42.520752 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:42.520757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:42.521822 (+ 1065us) server_negotiation.cc:407] Connection header received
0811 12:33:42.523643 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:42.523648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:42.523727 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:42.523822 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:42.525426 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:42.526177 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:42.527774 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:42.528232 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:42.528353 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:42.532470 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:42.532475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:42.532479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:42.532517 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:42.535074 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:42.536047 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:42.536052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:42.536054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:42.536208 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:42.536676 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:42.536679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:42.536681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:42.536963 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:42.537074 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:42.538061 (+ 987us) server_negotiation.cc:299] Negotiation successful
0811 12:33:42.538251 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":246,"threads_started":1}
I20260811 12:33:42.540033 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:42.520796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36079 (local address 127.3.204.65:60077)
0811 12:33:42.521685 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:42.521733 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:42.523384 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:42.524099 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:42.524112 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:42.524635 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:42.525248 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:42.525267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:42.526336 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:42.526340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:42.527618 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:42.527637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:42.528474 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:42.528483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:42.528728 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:42.529411 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:42.529435 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:42.532218 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:42.535237 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:42.535247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:42.535267 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:42.535917 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:42.536312 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:42.536317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:42.536321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:42.536581 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:42.537088 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:42.537097 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:42.537582 (+ 485us) client_negotiation.cc:769] Sending connection context
0811 12:33:42.537909 (+ 327us) client_negotiation.cc:240] Negotiation successful
0811 12:33:42.538527 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
I20260811 12:33:42.541786 10663 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36079
I20260811 12:33:42.542114 10663 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:42.542826 10663 heartbeater.cc:511] Master 127.3.204.126:36079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
I20260811 12:33:42.545260 10356 ts_manager.cc:194] Registered new tserver with Master: 2142db33557b41e59083daa274a1f283 (127.3.204.65:43787)
I20260811 12:33:42.547220 10356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:60077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:42.571475 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36079
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:42.903008 10674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:42.903301 10674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:42.903429 10674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:42.910734 10674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:42.910858 10674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:42.911021 10674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:42.920856 10674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:42.922204 10674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:42.923557 10674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:42.937950 10681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:42.937916 10684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:42.937980 10682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:42.938663 10674 server_base.cc:1034] running on GCE node
I20260811 12:33:42.939250 10674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:42.940215 10674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:42.941412 10674 hybrid_clock.cc:648] HybridClock initialized: now 1786451622941369 us; error 38 us; skew 500 ppm
I20260811 12:33:42.941710 10674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:42 dist-test-slave-lgr5 krb5kdc[10286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:42.944815 10674 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:42.946302 10674 webserver.cc:460] Webserver started at http://127.3.204.66:36229/ using document root <none> and password file <none>
I20260811 12:33:42.946990 10674 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:42.947103 10674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:42.947412 10674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:42.950428 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "fb3ef3e884a64fb6a8bc75cbf492f59e"
format_stamp: "Formatted at 2026-08-11 12:33:42 on dist-test-slave-lgr5"
server_key: "08785137cbbeef1c11ce8a5ff8b4c6a6"
server_key_iv: "1a33d049c5f56221bebf012fe8830511"
server_key_version: "encryptionkey@0"
I20260811 12:33:42.951205 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "fb3ef3e884a64fb6a8bc75cbf492f59e"
format_stamp: "Formatted at 2026-08-11 12:33:42 on dist-test-slave-lgr5"
server_key: "08785137cbbeef1c11ce8a5ff8b4c6a6"
server_key_iv: "1a33d049c5f56221bebf012fe8830511"
server_key_version: "encryptionkey@0"
I20260811 12:33:42.956576 10674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:33:42.960225 10694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:42.961578 10674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:33:42.961745 10674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "fb3ef3e884a64fb6a8bc75cbf492f59e"
format_stamp: "Formatted at 2026-08-11 12:33:42 on dist-test-slave-lgr5"
server_key: "08785137cbbeef1c11ce8a5ff8b4c6a6"
server_key_iv: "1a33d049c5f56221bebf012fe8830511"
server_key_version: "encryptionkey@0"
I20260811 12:33:42.961941 10674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:42.978204 10674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:42.982940 10674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:42.983284 10674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:42.984385 10674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:42.985803 10674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:42.985883 10674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:42.986018 10674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:42.986070 10674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:43.017930 10674 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:36471
I20260811 12:33:43.017998 10885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:36471 every 8 connection(s)
I20260811 12:33:43.019258 10674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:43.023458 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10674
I20260811 12:33:43.023674 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:43.024235 3889 external_mini_cluster.cc:1428] Setting key 22527b1de194c5363be4a075d29eec8c
Aug 11 12:33:43 dist-test-slave-lgr5 krb5kdc[10286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:43.042872 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.022311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:47695 (local address 127.3.204.126:36079)
0811 12:33:43.022513 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:43.022518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:43.023718 (+ 1200us) server_negotiation.cc:407] Connection header received
0811 12:33:43.025833 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:43.025839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:43.025926 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:43.026032 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:43.028182 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.028923 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.031046 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.031597 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.031700 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.036018 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:43.036025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:43.036028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:43.036064 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:43.039388 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.040276 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.040281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.040283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.040422 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.040890 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.040894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.040897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.041202 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:43.041342 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:43.042205 (+ 863us) server_negotiation.cc:299] Negotiation successful
0811 12:33:43.042534 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:33:43.043905 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.022768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36079 (local address 127.3.204.66:47695)
0811 12:33:43.023525 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:43.023592 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:43.025543 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:43.026320 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:43.026333 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:43.027129 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:43.027963 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:43.027991 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.029100 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.029105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:43.030901 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:43.030928 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.031828 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.031835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:43.032166 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.032899 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:43.032938 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:43.035754 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:43.039577 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:43.039587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:43.039603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:43.040136 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:43.040541 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:43.040547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:43.040551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:43.040764 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:43.041339 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:43.041348 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:43.041794 (+ 446us) client_negotiation.cc:769] Sending connection context
0811 12:33:43.042055 (+ 261us) client_negotiation.cc:240] Negotiation successful
0811 12:33:43.042689 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":311,"threads_started":1}
I20260811 12:33:43.045555 10886 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36079
I20260811 12:33:43.045883 10886 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
I20260811 12:33:43.046772 10886 heartbeater.cc:511] Master 127.3.204.126:36079 requested a full tablet report, sending...
I20260811 12:33:43.048787 10356 ts_manager.cc:194] Registered new tserver with Master: fb3ef3e884a64fb6a8bc75cbf492f59e (127.3.204.66:36471)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
I20260811 12:33:43.049768 10356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:47695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:43.075753 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36079
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:43.413872 10894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:43.414188 10894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:43.414332 10894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:43.421708 10894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:43.421849 10894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:43.422012 10894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:43.432107 10894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:43.433514 10894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:43.434888 10894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:43.448503 10906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:43.448503 10908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:43.448654 10911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:43.449409 10894 server_base.cc:1034] running on GCE node
I20260811 12:33:43.450011 10894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:43.450953 10894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:43.452162 10894 hybrid_clock.cc:648] HybridClock initialized: now 1786451623452097 us; error 56 us; skew 500 ppm
I20260811 12:33:43.452488 10894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:43 dist-test-slave-lgr5 krb5kdc[10286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:43.455684 10894 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:43.457238 10894 webserver.cc:460] Webserver started at http://127.3.204.67:34007/ using document root <none> and password file <none>
I20260811 12:33:43.457955 10894 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:43.458086 10894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:43.458400 10894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:43.461452 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "fc560d6a40984f3eae84d72a9d17dae1"
format_stamp: "Formatted at 2026-08-11 12:33:43 on dist-test-slave-lgr5"
server_key: "6212307b14580f344395fce4657ffc83"
server_key_iv: "69c584f035f3ae7e2d691ad8d07082d9"
server_key_version: "encryptionkey@0"
I20260811 12:33:43.462283 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "fc560d6a40984f3eae84d72a9d17dae1"
format_stamp: "Formatted at 2026-08-11 12:33:43 on dist-test-slave-lgr5"
server_key: "6212307b14580f344395fce4657ffc83"
server_key_iv: "69c584f035f3ae7e2d691ad8d07082d9"
server_key_version: "encryptionkey@0"
I20260811 12:33:43.468340 10894 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260811 12:33:43.472627 10922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:43.474133 10894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:33:43.474318 10894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "fc560d6a40984f3eae84d72a9d17dae1"
format_stamp: "Formatted at 2026-08-11 12:33:43 on dist-test-slave-lgr5"
server_key: "6212307b14580f344395fce4657ffc83"
server_key_iv: "69c584f035f3ae7e2d691ad8d07082d9"
server_key_version: "encryptionkey@0"
I20260811 12:33:43.474555 10894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:43.493613 10894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:43.498289 10894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:43.498582 10894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:43.499621 10894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:43.501159 10894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:43.501236 10894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:43.501359 10894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:43.501410 10894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:43.533952 10894 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:37781
I20260811 12:33:43.534029 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:37781 every 8 connection(s)
I20260811 12:33:43.535347 10894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:43.538657 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10894
I20260811 12:33:43.538831 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:43.539402 3889 external_mini_cluster.cc:1428] Setting key 48381a513e72251e69bfd6ce4f55d6a9
Aug 11 12:33:43 dist-test-slave-lgr5 krb5kdc[10286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:43.550489 10663 heartbeater.cc:503] Master 127.3.204.126:36079 was elected leader, sending a full tablet report...
I20260811 12:33:43.556883 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.538139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:56491 (local address 127.3.204.126:36079)
0811 12:33:43.538361 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:43.538367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:43.539696 (+ 1329us) server_negotiation.cc:407] Connection header received
0811 12:33:43.542059 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:43.542065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:43.542153 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:43.542265 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:43.543922 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.544799 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.546334 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.546763 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.546871 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.550384 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:43.550389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:43.550393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:43.550430 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:43.553609 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.554459 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.554466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.554469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.554611 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.555072 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.555076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.555078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.555364 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:43.555520 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:43.556283 (+ 763us) server_negotiation.cc:299] Negotiation successful
0811 12:33:43.556531 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:33:43.557893 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.538629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36079 (local address 127.3.204.67:56491)
0811 12:33:43.539494 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:43.539552 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:43.541784 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:43.542507 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:43.542519 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:43.543171 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:43.543752 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:43.543770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.544939 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.544943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:43.546178 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:43.546190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.546969 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.546974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:43.547183 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.547830 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:43.547861 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:43.550140 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:43.553759 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:43.553770 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:43.553788 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:43.554333 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:43.554725 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:43.554730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:43.554734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:43.554948 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:43.555500 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:43.555509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:43.555870 (+ 361us) client_negotiation.cc:769] Sending connection context
0811 12:33:43.556195 (+ 325us) client_negotiation.cc:240] Negotiation successful
0811 12:33:43.556769 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":305,"threads_started":1}
I20260811 12:33:43.559643 11107 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36079
I20260811 12:33:43.559973 11107 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:43.560779 11107 heartbeater.cc:511] Master 127.3.204.126:36079 requested a full tablet report, sending...
I20260811 12:33:43.562366 10356 ts_manager.cc:194] Registered new tserver with Master: fc560d6a40984f3eae84d72a9d17dae1 (127.3.204.67:37781)
I20260811 12:33:43.563210 10356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:56491
I20260811 12:33:43.567775 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 12:33:43 dist-test-slave-lgr5 krb5kdc[10286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 12:33:43 dist-test-slave-lgr5 krb5kdc[10286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:43.620585 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.605162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.3.204.65:43787)
0811 12:33:43.605851 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:43.605858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:43.605891 (+ 33us) server_negotiation.cc:407] Connection header received
0811 12:33:43.605956 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:43.605963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:43.606198 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:43.606350 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:43.607806 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.608922 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.610154 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.610732 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.610850 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.614792 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:43.614806 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:43.614824 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:43.614867 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:43.617746 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.618426 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.618434 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.618437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.618575 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.618951 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.618955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.618958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.619337 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:43.619629 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:43.619947 (+ 318us) server_negotiation.cc:299] Negotiation successful
0811 12:33:43.620245 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":316,"threads_started":1}
Aug 11 12:33:43 dist-test-slave-lgr5 krb5kdc[10286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:43.638645 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.625317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.3.204.126:36079)
0811 12:33:43.625551 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:43.625557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:43.625659 (+ 102us) server_negotiation.cc:407] Connection header received
0811 12:33:43.625796 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:43.625799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:43.625867 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:43.625949 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:43.627260 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.628020 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.629232 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.629694 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.629798 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.633554 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:43.633559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:43.633562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:43.633599 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:43.635917 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.636626 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.636631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.636633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.636779 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.637136 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.637140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.637142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.637497 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:43.637758 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:43.638046 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 12:33:43.638271 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260811 12:33:43.639276 10356 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52402
I20260811 12:33:43.640303 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10452
I20260811 12:33:43.652869 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.641087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.3.204.65:43787)
0811 12:33:43.641303 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:43.641310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:43.641496 (+ 186us) server_negotiation.cc:407] Connection header received
0811 12:33:43.641673 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:43.641677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:43.641742 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:43.641832 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:43.643321 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.644008 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.645198 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.645639 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.645745 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.647951 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:43.647955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:43.647958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:43.647986 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:43.650455 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.651128 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.651132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.651135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.651267 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.651639 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.651642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.651644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.651897 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:43.652110 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:43.652456 (+ 346us) server_negotiation.cc:299] Negotiation successful
0811 12:33:43.652635 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 12:33:43.653484 10654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:43.736198 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10452
I20260811 12:33:43.761579 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10674
Aug 11 12:33:43 dist-test-slave-lgr5 krb5kdc[10286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:33:43.777326 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.762484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.3.204.66:36471)
0811 12:33:43.763047 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:43.763054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:43.763080 (+ 26us) server_negotiation.cc:407] Connection header received
0811 12:33:43.763143 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:43.763150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:43.763377 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:43.763512 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:43.764719 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.765880 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.767268 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.767826 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.767925 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.772002 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:43.772015 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:43.772033 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:43.772070 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:43.774766 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.775475 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.775482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.775485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.775612 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.775893 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.775896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.775898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.776311 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:43.776549 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:43.776810 (+ 261us) server_negotiation.cc:299] Negotiation successful
0811 12:33:43.777030 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":267,"threads_started":1}
I20260811 12:33:43.778694 10881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:43.872584 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10674
I20260811 12:33:43.906561 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10894
Aug 11 12:33:43 dist-test-slave-lgr5 krb5kdc[10286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:33:43.923152 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:43.907558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.3.204.67:37781)
0811 12:33:43.908268 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:43.908275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:43.908304 (+ 29us) server_negotiation.cc:407] Connection header received
0811 12:33:43.908370 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:43.908376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:43.908581 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:43.908713 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:43.910022 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.911064 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.912370 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:43.912923 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:43.913031 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:43.917252 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:43.917268 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:43.917296 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:43.917354 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:43.920510 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.921272 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.921280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.921282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.921424 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:43.921746 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:43.921749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:43.921750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:43.922123 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:43.922354 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:43.922627 (+ 273us) server_negotiation.cc:299] Negotiation successful
0811 12:33:43.922846 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":360,"threads_started":1}
I20260811 12:33:43.924504 11101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:44.010061 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10894
I20260811 12:33:44.032539 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10313
I20260811 12:33:44.033690 10399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:44.138616 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10313
2026-08-11T12:33:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4154 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:44 dist-test-slave-lgr5 krb5kdc[11141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:44 dist-test-slave-lgr5 krb5kdc[11141](info): set up 2 sockets
Aug 11 12:33:44 dist-test-slave-lgr5 krb5kdc[11141](info): commencing operation
krb5kdc: starting...
W20260811 12:33:45.388813 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:45 dist-test-slave-lgr5 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:45.612332 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:34973
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:34973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:45.948490 11167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:45.948783 11167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:45.948885 11167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:45.956149 11167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:45.956265 11167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:45.956318 11167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:45.956374 11167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:45.956421 11167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:45.966440 11167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:34973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:45.967757 11167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:45.969046 11167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:45.981891 11175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:45.981825 11180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:45.981868 11178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:45.982507 11167 server_base.cc:1034] running on GCE node
I20260811 12:33:45.983414 11167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:45.984994 11167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:45.986208 11167 hybrid_clock.cc:648] HybridClock initialized: now 1786451625986163 us; error 43 us; skew 500 ppm
I20260811 12:33:45.986585 11167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:45 dist-test-slave-lgr5 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:45.989681 11167 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:45.990938 11167 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:45.991053 11167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:45.991389 11167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:45.995117 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "e4a11c47b26540238e03c5370b2b4383"
format_stamp: "Formatted at 2026-08-11 12:33:45 on dist-test-slave-lgr5"
server_key: "a8f58f55b968b53111b47c12eafd3df7"
server_key_iv: "361f662b831bffdb1ac16a71b31a4867"
server_key_version: "encryptionkey@0"
I20260811 12:33:45.996233 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "e4a11c47b26540238e03c5370b2b4383"
format_stamp: "Formatted at 2026-08-11 12:33:45 on dist-test-slave-lgr5"
server_key: "a8f58f55b968b53111b47c12eafd3df7"
server_key_iv: "361f662b831bffdb1ac16a71b31a4867"
server_key_version: "encryptionkey@0"
I20260811 12:33:46.002965 11167 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260811 12:33:46.007279 11189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:46.008780 11167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:46.008944 11167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "e4a11c47b26540238e03c5370b2b4383"
format_stamp: "Formatted at 2026-08-11 12:33:45 on dist-test-slave-lgr5"
server_key: "a8f58f55b968b53111b47c12eafd3df7"
server_key_iv: "361f662b831bffdb1ac16a71b31a4867"
server_key_version: "encryptionkey@0"
I20260811 12:33:46.009150 11167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:46.058053 11167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:46.062299 11167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:46.062546 11167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:46.080710 11167 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:34973
I20260811 12:33:46.080730 11285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:34973 every 8 connection(s)
I20260811 12:33:46.081668 11167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:46.085193 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11167
I20260811 12:33:46.085420 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:46.085670 11286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:46.086059 3889 external_mini_cluster.cc:1428] Setting key 82dfa57f93429f1b3b9e5638c0d717dd
I20260811 12:33:46.100816 11286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383: Bootstrap starting.
Aug 11 12:33:46 dist-test-slave-lgr5 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:46.105026 11286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:46.106478 11286 log.cc:824] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:46.110206 11286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383: No bootstrap required, opened a new log
I20260811 12:33:46.111315 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:46.088124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.3.204.126:34973)
0811 12:33:46.089016 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:46.089032 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:46.089089 (+ 57us) server_negotiation.cc:407] Connection header received
0811 12:33:46.090719 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:46.090752 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:46.091299 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:46.091758 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:46.093228 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:46.095270 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:46.096710 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:46.097891 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:46.098121 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:46.102321 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:46.102335 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:46.102357 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:46.102422 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:46.106349 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:46.107170 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:46.107183 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:46.107192 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:46.107390 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:46.107906 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:46.107912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:46.107933 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:46.108753 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:46.109106 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:46.109431 (+ 325us) server_negotiation.cc:299] Negotiation successful
0811 12:33:46.109903 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":348,"threads_started":1}
I20260811 12:33:46.120414 11286 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a11c47b26540238e03c5370b2b4383" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34973 } }
I20260811 12:33:46.120837 11286 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:46.120942 11286 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a11c47b26540238e03c5370b2b4383, State: Initialized, Role: FOLLOWER
I20260811 12:33:46.121589 11286 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a11c47b26540238e03c5370b2b4383" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34973 } }
I20260811 12:33:46.121809 11286 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:46.121942 11286 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:46.122107 11286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:46.124469 11286 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a11c47b26540238e03c5370b2b4383" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34973 } }
I20260811 12:33:46.125087 11286 leader_election.cc:302] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4a11c47b26540238e03c5370b2b4383; no voters:
I20260811 12:33:46.125782 11286 leader_election.cc:288] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:46.125955 11297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:46.126619 11297 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [term 1 LEADER]: Becoming Leader. State: Replica: e4a11c47b26540238e03c5370b2b4383, State: Running, Role: LEADER
I20260811 12:33:46.127198 11297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a11c47b26540238e03c5370b2b4383" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34973 } }
I20260811 12:33:46.127993 11286 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:46.130218 11300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4a11c47b26540238e03c5370b2b4383. Latest consensus state: current_term: 1 leader_uuid: "e4a11c47b26540238e03c5370b2b4383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a11c47b26540238e03c5370b2b4383" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34973 } } }
I20260811 12:33:46.130245 11299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4a11c47b26540238e03c5370b2b4383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a11c47b26540238e03c5370b2b4383" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34973 } } }
I20260811 12:33:46.130853 11299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:46.130849 11300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:46.133562 11306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:46.138994 11306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:46.147972 11306 catalog_manager.cc:1347] Generated new cluster ID: 5b82ad0971494d2eaf6db88ae16346dd
I20260811 12:33:46.148144 11306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:46.166090 11306 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:46.168143 11306 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:46.190647 11306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383: Generated new TSK 0
I20260811 12:33:46.191874 11306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:46.247217 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34973
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:46.585866 11332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:46.586172 11332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:46.586295 11332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:46.593716 11332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:46.593837 11332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:46.593988 11332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:46.603930 11332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:46.605332 11332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:46.606653 11332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:46.619812 11344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:46.619822 11341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:46.619856 11347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:46.620747 11332 server_base.cc:1034] running on GCE node
I20260811 12:33:46.621423 11332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:46.622412 11332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:46.623631 11332 hybrid_clock.cc:648] HybridClock initialized: now 1786451626623578 us; error 46 us; skew 500 ppm
I20260811 12:33:46.623973 11332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:46 dist-test-slave-lgr5 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:46.627086 11332 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:46.628340 11332 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:46.628458 11332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:46.628767 11332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:46.632072 11332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "67d7d7f3fce744a79171ee6433152749"
format_stamp: "Formatted at 2026-08-11 12:33:46 on dist-test-slave-lgr5"
server_key: "fb3c847be7b6a5d4966742634771269d"
server_key_iv: "41c22fe8f725c7f7c986ae101656a11c"
server_key_version: "encryptionkey@0"
I20260811 12:33:46.632985 11332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "67d7d7f3fce744a79171ee6433152749"
format_stamp: "Formatted at 2026-08-11 12:33:46 on dist-test-slave-lgr5"
server_key: "fb3c847be7b6a5d4966742634771269d"
server_key_iv: "41c22fe8f725c7f7c986ae101656a11c"
server_key_version: "encryptionkey@0"
I20260811 12:33:46.638262 11332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:33:46.641889 11356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:46.643143 11332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:33:46.643301 11332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "67d7d7f3fce744a79171ee6433152749"
format_stamp: "Formatted at 2026-08-11 12:33:46 on dist-test-slave-lgr5"
server_key: "fb3c847be7b6a5d4966742634771269d"
server_key_iv: "41c22fe8f725c7f7c986ae101656a11c"
server_key_version: "encryptionkey@0"
I20260811 12:33:46.643522 11332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:46.664717 11332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:46.668772 11332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:46.669023 11332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:46.669986 11332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:46.671337 11332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:46.671424 11332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:46.671509 11332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:46.671562 11332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:46.702584 11332 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:36649
I20260811 12:33:46.702634 11532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:36649 every 8 connection(s)
I20260811 12:33:46.703459 11332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:46.709435 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11332
I20260811 12:33:46.709664 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:46.710337 3889 external_mini_cluster.cc:1428] Setting key d116ae51cd9c8ffebc4d68496d5b0cb7
Aug 11 12:33:46 dist-test-slave-lgr5 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:46.723726 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:46.705901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:55309 (local address 127.3.204.126:34973)
0811 12:33:46.706381 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:46.706388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:46.707315 (+ 927us) server_negotiation.cc:407] Connection header received
0811 12:33:46.709205 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:46.709211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:46.709307 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:46.709449 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:46.711217 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:46.712119 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:46.713762 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:46.714167 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:46.714282 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:46.717796 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:46.717801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:46.717804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:46.717840 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:46.720652 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:46.721388 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:46.721394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:46.721397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:46.721532 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:46.721950 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:46.721954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:46.721956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:46.722234 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:46.722368 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:46.723108 (+ 740us) server_negotiation.cc:299] Negotiation successful
0811 12:33:46.723347 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":219,"threads_started":1}
I20260811 12:33:46.724833 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:46.706448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34973 (local address 127.3.204.65:55309)
0811 12:33:46.707160 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:46.707199 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:46.708898 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:46.709649 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:46.709658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:46.710310 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:46.711009 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:46.711040 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:46.712283 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:46.712288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:46.713600 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:46.713614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:46.714399 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:46.714407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:46.714640 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:46.715331 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:46.715354 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:46.717569 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:46.720791 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:46.720801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:46.720819 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:46.721269 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:46.721640 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:46.721645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:46.721649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:46.721839 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:46.722369 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:46.722378 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:46.722728 (+ 350us) client_negotiation.cc:769] Sending connection context
0811 12:33:46.722993 (+ 265us) client_negotiation.cc:240] Negotiation successful
0811 12:33:46.723575 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":278,"threads_started":1}
I20260811 12:33:46.726560 11533 heartbeater.cc:347] Connected to a master server at 127.3.204.126:34973
I20260811 12:33:46.726871 11533 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:46.727553 11533 heartbeater.cc:511] Master 127.3.204.126:34973 requested a full tablet report, sending...
I20260811 12:33:46.729882 11214 ts_manager.cc:194] Registered new tserver with Master: 67d7d7f3fce744a79171ee6433152749 (127.3.204.65:36649)
I20260811 12:33:46.731387 11214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:55309
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:46.761790 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34973
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:47.105710 11542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:47.106069 11542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:47.106220 11542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:47.113950 11542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:47.114092 11542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:47.114256 11542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:47.124450 11542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:47.125898 11542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:47.127254 11542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:47.140563 11549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:47.140489 11550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:47.140522 11553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:47.141605 11542 server_base.cc:1034] running on GCE node
I20260811 12:33:47.142247 11542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:47.143224 11542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:47.144407 11542 hybrid_clock.cc:648] HybridClock initialized: now 1786451627144334 us; error 65 us; skew 500 ppm
I20260811 12:33:47.144753 11542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:47 dist-test-slave-lgr5 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:47.153687 11542 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:47.155061 11542 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:47.155198 11542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:47.155634 11542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:47.158866 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "112b7fed6daa4b7390f0fb4484f7ee9c"
format_stamp: "Formatted at 2026-08-11 12:33:47 on dist-test-slave-lgr5"
server_key: "e0bdfe9c18fb8af810d1ae3bc57d6422"
server_key_iv: "6bd929d38e2e7541410c371b0b7b71c9"
server_key_version: "encryptionkey@0"
I20260811 12:33:47.159734 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "112b7fed6daa4b7390f0fb4484f7ee9c"
format_stamp: "Formatted at 2026-08-11 12:33:47 on dist-test-slave-lgr5"
server_key: "e0bdfe9c18fb8af810d1ae3bc57d6422"
server_key_iv: "6bd929d38e2e7541410c371b0b7b71c9"
server_key_version: "encryptionkey@0"
I20260811 12:33:47.165365 11542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260811 12:33:47.169289 11561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:47.170691 11542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:33:47.170882 11542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "112b7fed6daa4b7390f0fb4484f7ee9c"
format_stamp: "Formatted at 2026-08-11 12:33:47 on dist-test-slave-lgr5"
server_key: "e0bdfe9c18fb8af810d1ae3bc57d6422"
server_key_iv: "6bd929d38e2e7541410c371b0b7b71c9"
server_key_version: "encryptionkey@0"
I20260811 12:33:47.171115 11542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:47.186780 11542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:47.191470 11542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:47.191759 11542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:47.192809 11542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:47.194235 11542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:47.194336 11542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:47.194453 11542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:47.194525 11542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:47.225476 11542 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:34247
I20260811 12:33:47.225590 11770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:34247 every 8 connection(s)
I20260811 12:33:47.226344 11542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:47.234019 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11542
I20260811 12:33:47.234210 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:47.234647 3889 external_mini_cluster.cc:1428] Setting key ca97d4b632d1a0d23afb8411ef574e08
Aug 11 12:33:47 dist-test-slave-lgr5 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:47.247182 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.228790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:49931 (local address 127.3.204.126:34973)
0811 12:33:47.229239 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:47.229243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:47.230156 (+ 913us) server_negotiation.cc:407] Connection header received
0811 12:33:47.231959 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:47.231965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:47.232052 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:47.232177 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:47.233715 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.234497 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.236481 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.237074 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.237205 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.240743 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:47.240749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:47.240753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:47.240794 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:47.243829 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:47.244694 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:47.244699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:47.244701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:47.244837 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:47.245343 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:47.245347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:47.245349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:47.245674 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:47.245790 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:47.246698 (+ 908us) server_negotiation.cc:299] Negotiation successful
0811 12:33:47.246888 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260811 12:33:47.248638 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.229270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34973 (local address 127.3.204.66:49931)
0811 12:33:47.230000 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:47.230039 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:47.231685 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:47.232417 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:47.232426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:47.232967 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:47.233533 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.233550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.234644 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.234648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.236327 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.236342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.237286 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.237291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.237505 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.238192 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:47.238221 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:47.240485 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:47.243988 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:47.243997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:47.244012 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:47.244576 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:47.244945 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:47.244950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:47.244954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:47.245203 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:47.245802 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:47.245810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:47.246258 (+ 448us) client_negotiation.cc:769] Sending connection context
0811 12:33:47.246581 (+ 323us) client_negotiation.cc:240] Negotiation successful
0811 12:33:47.247185 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":303,"threads_started":1}
I20260811 12:33:47.250104 11771 heartbeater.cc:347] Connected to a master server at 127.3.204.126:34973
I20260811 12:33:47.250432 11771 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:47.251281 11771 heartbeater.cc:511] Master 127.3.204.126:34973 requested a full tablet report, sending...
I20260811 12:33:47.252909 11214 ts_manager.cc:194] Registered new tserver with Master: 112b7fed6daa4b7390f0fb4484f7ee9c (127.3.204.66:34247)
I20260811 12:33:47.253736 11214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:49931
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:47.287699 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34973
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:47.621115 11777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:47.621415 11777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:47.621563 11777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:47.628935 11777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:47.629074 11777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:47.629232 11777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:47.639236 11777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:47.640636 11777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:47.641983 11777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:47.654726 11784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:47.654822 11783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:47.654850 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:47.655405 11777 server_base.cc:1034] running on GCE node
I20260811 12:33:47.656008 11777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:47.656982 11777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:47.658190 11777 hybrid_clock.cc:648] HybridClock initialized: now 1786451627658136 us; error 45 us; skew 500 ppm
I20260811 12:33:47.658501 11777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:47 dist-test-slave-lgr5 krb5kdc[11141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:47.661744 11777 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:47.662873 11777 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:47.662977 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:47.663287 11777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:47.666352 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "589acf03f9c24af0b0069780d29cc78c"
format_stamp: "Formatted at 2026-08-11 12:33:47 on dist-test-slave-lgr5"
server_key: "7e2836ff93470c61f0461fff82406809"
server_key_iv: "ec4f8c06635afc6b700c5ec8ee30b736"
server_key_version: "encryptionkey@0"
I20260811 12:33:47.667197 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "589acf03f9c24af0b0069780d29cc78c"
format_stamp: "Formatted at 2026-08-11 12:33:47 on dist-test-slave-lgr5"
server_key: "7e2836ff93470c61f0461fff82406809"
server_key_iv: "ec4f8c06635afc6b700c5ec8ee30b736"
server_key_version: "encryptionkey@0"
I20260811 12:33:47.672443 11777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:33:47.675940 11795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:47.677404 11777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:33:47.677583 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "589acf03f9c24af0b0069780d29cc78c"
format_stamp: "Formatted at 2026-08-11 12:33:47 on dist-test-slave-lgr5"
server_key: "7e2836ff93470c61f0461fff82406809"
server_key_iv: "ec4f8c06635afc6b700c5ec8ee30b736"
server_key_version: "encryptionkey@0"
I20260811 12:33:47.677770 11777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:47.710062 11777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:47.714684 11777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:47.714942 11777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:47.715938 11777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:47.717397 11777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:47.717489 11777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:47.717571 11777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:47.717628 11777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:47.734553 11533 heartbeater.cc:503] Master 127.3.204.126:34973 was elected leader, sending a full tablet report...
I20260811 12:33:47.749727 11777 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:35163
I20260811 12:33:47.749784 11975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:35163 every 8 connection(s)
I20260811 12:33:47.750602 11777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:47.759696 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11777
I20260811 12:33:47.759900 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:47.760365 3889 external_mini_cluster.cc:1428] Setting key 54021cd5b96d264bda6c35d5a86a4223
Aug 11 12:33:47 dist-test-slave-lgr5 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:47.770663 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.753026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:45105 (local address 127.3.204.126:34973)
0811 12:33:47.753456 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:47.753461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:47.754481 (+ 1020us) server_negotiation.cc:407] Connection header received
0811 12:33:47.756305 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:47.756309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:47.756398 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:47.756500 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:47.758060 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.758871 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.760649 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.761163 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.761301 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.765404 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:47.765411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:47.765415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:47.765451 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:47.767961 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:47.768690 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:47.768694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:47.768696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:47.768814 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:47.769159 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:47.769162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:47.769164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:47.769425 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:47.769543 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:47.770221 (+ 678us) server_negotiation.cc:299] Negotiation successful
0811 12:33:47.770390 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":204,"threads_started":1}
I20260811 12:33:47.771690 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.753623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34973 (local address 127.3.204.67:45105)
0811 12:33:47.754324 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:47.754362 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:47.755986 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:47.756687 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:47.756694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:47.757252 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:47.757830 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.757853 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.759015 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.759019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.760464 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.760482 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.761384 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.761390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.761672 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.762457 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:47.762492 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:47.765174 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:47.768123 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:47.768133 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:47.768149 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:47.768590 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:47.768890 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:47.768893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:47.768895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:47.769077 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:47.769537 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:47.769543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:47.769885 (+ 342us) client_negotiation.cc:769] Sending connection context
0811 12:33:47.770139 (+ 254us) client_negotiation.cc:240] Negotiation successful
0811 12:33:47.770540 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":277,"threads_started":1}
I20260811 12:33:47.773150 11976 heartbeater.cc:347] Connected to a master server at 127.3.204.126:34973
I20260811 12:33:47.773455 11976 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:47.774137 11976 heartbeater.cc:511] Master 127.3.204.126:34973 requested a full tablet report, sending...
I20260811 12:33:47.775715 11214 ts_manager.cc:194] Registered new tserver with Master: 589acf03f9c24af0b0069780d29cc78c (127.3.204.67:35163)
I20260811 12:33:47.776652 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:33:47.776723 11214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:45105
I20260811 12:33:47.793942 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.781452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.3.204.126:34973)
0811 12:33:47.781637 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:47.781642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:47.782184 (+ 542us) server_negotiation.cc:407] Connection header received
0811 12:33:47.782386 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:47.782390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:47.782474 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:47.782556 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:47.784071 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.784850 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.786116 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.786554 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.786688 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.788891 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:47.788897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:47.788900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:47.788932 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:47.791376 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:47.791993 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:47.791999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:47.792003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:47.792172 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:47.792522 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:47.792528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:47.792531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:47.792847 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:47.793067 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:47.793409 (+ 342us) server_negotiation.cc:299] Negotiation successful
0811 12:33:47.793645 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:33:47.798148 11214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:47.805154 11214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:47.842021 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.818171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:36649 (local address 127.0.0.1:38180)
0811 12:33:47.827051 (+ 8880us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:47.827099 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:47.827326 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:47.834132 (+ 6806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:47.834140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:47.834190 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:47.834634 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.834645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.836668 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.836673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.839114 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.839130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.841011 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.841018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.841227 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.841278 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 12:33:47.841396 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 12:33:47.841614 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8587,"thread_start_us":8432,"threads_started":1}
I20260811 12:33:47.842413 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.818321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.3.204.65:36649)
0811 12:33:47.833367 (+ 15046us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:47.833376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:47.833423 (+ 47us) server_negotiation.cc:407] Connection header received
0811 12:33:47.833553 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:47.833562 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:47.833850 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:47.834800 (+ 950us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:47.835054 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.836503 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.839273 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.840751 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.841567 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.841802 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:47.841888 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 12:33:47.842066 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14849,"thread_start_us":14677,"threads_started":1}
I20260811 12:33:47.842628 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.818859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.3.204.66:34247)
0811 12:33:47.832619 (+ 13760us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:47.832631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:47.833970 (+ 1339us) server_negotiation.cc:407] Connection header received
0811 12:33:47.834974 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:47.834982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:47.835234 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:47.835441 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:47.836074 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.837454 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.839555 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.840965 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.841100 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.841315 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:47.842122 (+ 807us) server_negotiation.cc:299] Negotiation successful
0811 12:33:47.842296 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13528,"thread_start_us":13771,"threads_started":1}
I20260811 12:33:47.842984 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.818581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:34247 (local address 127.0.0.1:53780)
0811 12:33:47.833817 (+ 15236us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:47.833841 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:47.834781 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:47.835449 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:47.835454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:47.835481 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:47.835860 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.835874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.837606 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.837611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.839377 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.839392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.841478 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.841485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.841652 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.841680 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 12:33:47.841774 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 12:33:47.841912 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15084,"thread_start_us":14904,"threads_started":1}
I20260811 12:33:47.843288 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.818607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:35163 (local address 127.0.0.1:55918)
0811 12:33:47.834479 (+ 15872us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:47.834500 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:47.834641 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:47.835944 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:47.835949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:47.835978 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:47.836665 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.836676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.838886 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.838892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.840615 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:47.840628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.842442 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.842449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:47.842622 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.842655 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 12:33:47.842724 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 12:33:47.842851 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15733,"thread_start_us":15569,"threads_started":1}
I20260811 12:33:47.844579 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:47.819283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.3.204.67:35163)
0811 12:33:47.835356 (+ 16073us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:47.835362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:47.835396 (+ 34us) server_negotiation.cc:407] Connection header received
0811 12:33:47.835498 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:47.835505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:47.835725 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:47.835924 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:47.836945 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.838720 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.840802 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:47.842210 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:47.843063 (+ 853us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:47.843288 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:47.843379 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 12:33:47.843542 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15913,"thread_start_us":15755,"threads_started":1}
I20260811 12:33:47.846784 11421 tablet_service.cc:1467] Processing CreateTablet for tablet 8821f6a1879741b6919deea99c8301b5 (DEFAULT_TABLE table=test-table [id=5347b200b3d34624bd592253cab410a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:47.846784 11649 tablet_service.cc:1467] Processing CreateTablet for tablet 8821f6a1879741b6919deea99c8301b5 (DEFAULT_TABLE table=test-table [id=5347b200b3d34624bd592253cab410a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:47.847856 11870 tablet_service.cc:1467] Processing CreateTablet for tablet 8821f6a1879741b6919deea99c8301b5 (DEFAULT_TABLE table=test-table [id=5347b200b3d34624bd592253cab410a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:47.848348 11649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8821f6a1879741b6919deea99c8301b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:47.848352 11421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8821f6a1879741b6919deea99c8301b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:47.849165 11870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8821f6a1879741b6919deea99c8301b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:47.860321 12002 tablet_bootstrap.cc:492] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Bootstrap starting.
I20260811 12:33:47.862845 12004 tablet_bootstrap.cc:492] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Bootstrap starting.
I20260811 12:33:47.863700 12003 tablet_bootstrap.cc:492] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Bootstrap starting.
I20260811 12:33:47.864773 12002 tablet_bootstrap.cc:654] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:47.866197 12002 log.cc:824] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:47.867434 12003 tablet_bootstrap.cc:654] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:47.867699 12004 tablet_bootstrap.cc:654] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:47.868750 12004 log.cc:824] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:47.868778 12003 log.cc:824] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:47.869825 12002 tablet_bootstrap.cc:492] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: No bootstrap required, opened a new log
I20260811 12:33:47.870146 12002 ts_tablet_manager.cc:1397] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260811 12:33:47.871582 12003 tablet_bootstrap.cc:492] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: No bootstrap required, opened a new log
I20260811 12:33:47.871829 12004 tablet_bootstrap.cc:492] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: No bootstrap required, opened a new log
I20260811 12:33:47.871842 12003 ts_tablet_manager.cc:1397] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260811 12:33:47.872129 12004 ts_tablet_manager.cc:1397] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260811 12:33:47.880396 12003 raft_consensus.cc:348] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:47.880805 12003 raft_consensus.cc:374] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:47.880931 12003 raft_consensus.cc:729] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d7d7f3fce744a79171ee6433152749, State: Initialized, Role: FOLLOWER
I20260811 12:33:47.880967 12002 raft_consensus.cc:348] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:47.881501 12002 raft_consensus.cc:374] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:47.881613 12002 raft_consensus.cc:729] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589acf03f9c24af0b0069780d29cc78c, State: Initialized, Role: FOLLOWER
I20260811 12:33:47.881573 12003 consensus_queue.cc:260] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:47.882872 12004 raft_consensus.cc:348] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:47.883373 12004 raft_consensus.cc:374] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:47.883402 12003 ts_tablet_manager.cc:1428] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260811 12:33:47.883548 12004 raft_consensus.cc:729] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 112b7fed6daa4b7390f0fb4484f7ee9c, State: Initialized, Role: FOLLOWER
I20260811 12:33:47.882341 12002 consensus_queue.cc:260] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:47.884218 12004 consensus_queue.cc:260] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:47.888481 11771 heartbeater.cc:503] Master 127.3.204.126:34973 was elected leader, sending a full tablet report...
I20260811 12:33:47.889626 12004 ts_tablet_manager.cc:1428] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260811 12:33:47.890729 11976 heartbeater.cc:503] Master 127.3.204.126:34973 was elected leader, sending a full tablet report...
I20260811 12:33:47.891237 12002 ts_tablet_manager.cc:1428] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
W20260811 12:33:47.955719 11534 tablet.cc:2369] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:33:47.978235 11772 tablet.cc:2369] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:33:48.002063 11977 tablet.cc:2369] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:33:48.111865 12012 raft_consensus.cc:482] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:48.112264 12012 raft_consensus.cc:504] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:48.114128 12012 leader_election.cc:288] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589acf03f9c24af0b0069780d29cc78c (127.3.204.67:35163), 112b7fed6daa4b7390f0fb4484f7ee9c (127.3.204.66:34247)
I20260811 12:33:48.123524 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:48.114709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:45217 (local address 127.3.204.67:35163)
0811 12:33:48.114926 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:48.114932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:48.115778 (+ 846us) server_negotiation.cc:407] Connection header received
0811 12:33:48.115986 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:48.115992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:48.116102 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:48.116203 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:48.117057 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.118191 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.121086 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.122301 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.122505 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:48.122569 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:48.123101 (+ 532us) server_negotiation.cc:299] Negotiation successful
0811 12:33:48.123248 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 12:33:48.123528 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:48.114746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:59715 (local address 127.3.204.66:34247)
0811 12:33:48.114925 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:48.114932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:48.115774 (+ 842us) server_negotiation.cc:407] Connection header received
0811 12:33:48.115986 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:48.115992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:48.116103 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:48.116206 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:33:48.117057 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.118191 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.121087 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.122301 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.122468 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:48.122550 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:48.123101 (+ 551us) server_negotiation.cc:299] Negotiation successful
0811 12:33:48.123248 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:33:48.123780 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:48.114636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:35163 (local address 127.3.204.65:45217)
0811 12:33:48.115613 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:48.115627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:48.115798 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:48.116229 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:48.116234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:48.116278 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:48.116810 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:48.116828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.118329 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.118333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:48.120891 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:48.120909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.122592 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.122598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:48.122803 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:48.122916 (+ 113us) client_negotiation.cc:769] Sending connection context
0811 12:33:48.123018 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 12:33:48.123224 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":278,"spinlock_wait_cycles":4864,"thread_start_us":496,"threads_started":1}
I20260811 12:33:48.123800 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:48.114482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:34247 (local address 127.3.204.65:59715)
0811 12:33:48.115597 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:48.115623 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:48.115834 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:48.116229 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:48.116234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:48.116270 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:33:48.116846 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:48.116857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.118327 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.118333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:48.120891 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:48.120909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.122579 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.122586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:48.122802 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:48.122913 (+ 111us) client_negotiation.cc:769] Sending connection context
0811 12:33:48.123018 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 12:33:48.123224 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":174,"spinlock_wait_cycles":44160,"thread_start_us":653,"threads_started":1}
I20260811 12:33:48.125185 11684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8821f6a1879741b6919deea99c8301b5" candidate_uuid: "67d7d7f3fce744a79171ee6433152749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" is_pre_election: true
I20260811 12:33:48.125185 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8821f6a1879741b6919deea99c8301b5" candidate_uuid: "67d7d7f3fce744a79171ee6433152749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589acf03f9c24af0b0069780d29cc78c" is_pre_election: true
I20260811 12:33:48.125828 11901 raft_consensus.cc:2454] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d7d7f3fce744a79171ee6433152749 in term 0.
I20260811 12:33:48.125828 11684 raft_consensus.cc:2454] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d7d7f3fce744a79171ee6433152749 in term 0.
I20260811 12:33:48.126669 11358 leader_election.cc:302] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589acf03f9c24af0b0069780d29cc78c, 67d7d7f3fce744a79171ee6433152749; no voters:
I20260811 12:33:48.127120 12012 raft_consensus.cc:2781] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:48.127271 12012 raft_consensus.cc:482] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:48.127383 12012 raft_consensus.cc:3037] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:48.130093 12012 raft_consensus.cc:504] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:48.130720 12012 leader_election.cc:288] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [CANDIDATE]: Term 1 election: Requested vote from peers 589acf03f9c24af0b0069780d29cc78c (127.3.204.67:35163), 112b7fed6daa4b7390f0fb4484f7ee9c (127.3.204.66:34247)
I20260811 12:33:48.131311 11684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8821f6a1879741b6919deea99c8301b5" candidate_uuid: "67d7d7f3fce744a79171ee6433152749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c"
I20260811 12:33:48.131311 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8821f6a1879741b6919deea99c8301b5" candidate_uuid: "67d7d7f3fce744a79171ee6433152749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589acf03f9c24af0b0069780d29cc78c"
I20260811 12:33:48.131560 11901 raft_consensus.cc:3037] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:48.131554 11684 raft_consensus.cc:3037] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:48.134851 11901 raft_consensus.cc:2454] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d7d7f3fce744a79171ee6433152749 in term 1.
I20260811 12:33:48.134851 11684 raft_consensus.cc:2454] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d7d7f3fce744a79171ee6433152749 in term 1.
I20260811 12:33:48.135391 11358 leader_election.cc:302] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 112b7fed6daa4b7390f0fb4484f7ee9c, 589acf03f9c24af0b0069780d29cc78c, 67d7d7f3fce744a79171ee6433152749; no voters:
I20260811 12:33:48.135751 12012 raft_consensus.cc:2781] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:48.136255 12012 raft_consensus.cc:686] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 1 LEADER]: Becoming Leader. State: Replica: 67d7d7f3fce744a79171ee6433152749, State: Running, Role: LEADER
I20260811 12:33:48.137100 12012 consensus_queue.cc:237] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } }
I20260811 12:33:48.142300 11213 catalog_manager.cc:5495] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67d7d7f3fce744a79171ee6433152749 (127.3.204.65). New cstate: current_term: 1 leader_uuid: "67d7d7f3fce744a79171ee6433152749" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d7d7f3fce744a79171ee6433152749" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 36649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20260811 12:33:48.212821 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:48.207258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.3.204.65:36649)
0811 12:33:48.207490 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:48.207495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:48.207515 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:33:48.207773 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:48.207778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:48.207881 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:48.208025 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:33:48.208523 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.209254 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.210546 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.210955 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.211051 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:48.211565 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:33:48.212182 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:33:48.212305 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:48.212477 (+ 172us) server_negotiation.cc:299] Negotiation successful
0811 12:33:48.212550 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 12:33:48.221186 11901 raft_consensus.cc:1261] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 1 FOLLOWER]: Refusing update from remote peer 67d7d7f3fce744a79171ee6433152749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:48.221186 11684 raft_consensus.cc:1261] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 1 FOLLOWER]: Refusing update from remote peer 67d7d7f3fce744a79171ee6433152749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:48.222168 12020 consensus_queue.cc:1035] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "112b7fed6daa4b7390f0fb4484f7ee9c" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:48.222497 12012 consensus_queue.cc:1035] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "589acf03f9c24af0b0069780d29cc78c" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:48.238091 12024 mvcc.cc:205] Tried to move back new op lower bound from 7317305869184385024 to 7317305868853665792. Current Snapshot: MvccSnapshot[applied={T|T < 7317305869184385024}]
I20260811 12:33:48.249442 12025 mvcc.cc:205] Tried to move back new op lower bound from 7317305869184385024 to 7317305868853665792. Current Snapshot: MvccSnapshot[applied={T|T < 7317305869184385024}]
I20260811 12:33:48.263286 11214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:48.263597 11214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:48.267705 11214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4a11c47b26540238e03c5370b2b4383: Sending DeleteTablet for 3 replicas of tablet 8821f6a1879741b6919deea99c8301b5
I20260811 12:33:48.268997 11870 tablet_service.cc:1514] Processing DeleteTablet for tablet 8821f6a1879741b6919deea99c8301b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:48 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:55918
I20260811 12:33:48.269315 11421 tablet_service.cc:1514] Processing DeleteTablet for tablet 8821f6a1879741b6919deea99c8301b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:48 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:38180
I20260811 12:33:48.269341 11649 tablet_service.cc:1514] Processing DeleteTablet for tablet 8821f6a1879741b6919deea99c8301b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:48 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:53780
I20260811 12:33:48.271528 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11332
I20260811 12:33:48.282048 12038 tablet_replica.cc:330] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: stopping tablet replica
I20260811 12:33:48.282729 12038 raft_consensus.cc:2229] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:48.283428 12038 raft_consensus.cc:2258] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:48.285948 12038 ts_tablet_manager.cc:1905] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:48.293484 12038 ts_tablet_manager.cc:1918] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:33:48.293776 12038 log.cc:1196] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/8821f6a1879741b6919deea99c8301b5
I20260811 12:33:48.294414 12038 ts_tablet_manager.cc:1939] T 8821f6a1879741b6919deea99c8301b5 P 589acf03f9c24af0b0069780d29cc78c: Deleting consensus metadata
I20260811 12:33:48.296766 11191 catalog_manager.cc:4841] TS 589acf03f9c24af0b0069780d29cc78c (127.3.204.67:35163): tablet 8821f6a1879741b6919deea99c8301b5 (table test-table [id=5347b200b3d34624bd592253cab410a8]) successfully deleted
I20260811 12:33:48.298607 12039 tablet_replica.cc:330] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: stopping tablet replica
I20260811 12:33:48.299132 12039 raft_consensus.cc:2229] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:33:48.299928 12039 raft_consensus.cc:2258] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:48.301844 12041 tablet_replica.cc:330] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: stopping tablet replica
I20260811 12:33:48.302310 12041 raft_consensus.cc:2229] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:48.302835 12041 raft_consensus.cc:2258] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:48.305109 12039 ts_tablet_manager.cc:1905] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:48.305433 12041 ts_tablet_manager.cc:1905] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 12:33:48 dist-test-slave-lgr5 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:48.312736 12039 ts_tablet_manager.cc:1918] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:33:48.312858 12041 ts_tablet_manager.cc:1918] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:33:48.312966 12039 log.cc:1196] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/8821f6a1879741b6919deea99c8301b5
I20260811 12:33:48.313027 12041 log.cc:1196] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/8821f6a1879741b6919deea99c8301b5
I20260811 12:33:48.313616 12041 ts_tablet_manager.cc:1939] T 8821f6a1879741b6919deea99c8301b5 P 112b7fed6daa4b7390f0fb4484f7ee9c: Deleting consensus metadata
I20260811 12:33:48.313637 12039 ts_tablet_manager.cc:1939] T 8821f6a1879741b6919deea99c8301b5 P 67d7d7f3fce744a79171ee6433152749: Deleting consensus metadata
I20260811 12:33:48.315524 11190 catalog_manager.cc:4841] TS 67d7d7f3fce744a79171ee6433152749 (127.3.204.65:36649): tablet 8821f6a1879741b6919deea99c8301b5 (table test-table [id=5347b200b3d34624bd592253cab410a8]) successfully deleted
I20260811 12:33:48.315503 11192 catalog_manager.cc:4841] TS 112b7fed6daa4b7390f0fb4484f7ee9c (127.3.204.66:34247): tablet 8821f6a1879741b6919deea99c8301b5 (table test-table [id=5347b200b3d34624bd592253cab410a8]) successfully deleted
I20260811 12:33:48.317749 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:48.272394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.3.204.65:36649)
0811 12:33:48.272578 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:48.272585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:48.297138 (+ 24553us) server_negotiation.cc:407] Connection header received
0811 12:33:48.297229 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:48.297236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:48.297335 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:48.297454 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:48.299834 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.301093 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.303222 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.304117 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.304264 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:48.309664 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:48.309680 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:48.309699 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:48.309754 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:48.314142 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:48.315183 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:48.315194 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:48.315198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:48.315383 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:48.315894 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:48.315899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:48.315902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:48.316396 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:48.316708 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:48.317275 (+ 567us) server_negotiation.cc:299] Negotiation successful
0811 12:33:48.317478 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 12:33:48.318558 11524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:48.418429 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11332
I20260811 12:33:48.444146 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11542
Aug 11 12:33:48 dist-test-slave-lgr5 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:33:48.459486 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:48.444893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.3.204.66:34247)
0811 12:33:48.445060 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:48.445065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:48.445087 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:33:48.445248 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:48.445252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:48.445332 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:48.445417 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:48.446679 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.447454 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.448898 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.449299 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.449405 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:48.453718 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:48.453735 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:48.453754 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:48.453793 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:48.456688 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:48.457379 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:48.457386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:48.457389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:48.457560 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:48.457948 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:48.457953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:48.457955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:48.458387 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:48.458624 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:48.458992 (+ 368us) server_negotiation.cc:299] Negotiation successful
0811 12:33:48.459182 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 12:33:48.460397 11765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:48.550513 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11542
I20260811 12:33:48.575435 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11777
Aug 11 12:33:48 dist-test-slave-lgr5 krb5kdc[11141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:33:48.596758 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:48.576456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.3.204.67:35163)
0811 12:33:48.576649 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:48.576654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:48.576676 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:33:48.576778 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:48.576783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:48.576851 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:48.576956 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:48.578301 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.579107 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.580404 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:48.580994 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:48.581108 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:48.590946 (+ 9838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:48.590964 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:48.590981 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:48.591024 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:48.593838 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:48.594537 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:48.594545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:48.594547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:48.594706 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:48.595114 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:48.595117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:48.595119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:48.595536 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:48.595848 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:48.596196 (+ 348us) server_negotiation.cc:299] Negotiation successful
0811 12:33:48.596443 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 12:33:48.597492 11968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:48.687193 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11777
I20260811 12:33:48.711800 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11167
I20260811 12:33:48.713160 11274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:48.823599 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11167
2026-08-11T12:33:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4688 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-11T12:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:48Z Disabled control of system clock
I20260811 12:33:48.904979 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:38779
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 12:33:49.246032 12061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:49.246352 12061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:49.246474 12061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:49.253844 12061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:49.253963 12061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:49.254007 12061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:49.254087 12061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:49.254144 12061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:49.263600 12061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:38779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:49.265033 12061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:49.266911 12061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:49.279054 12072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:49.279090 12067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:49.279770 12068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:49.280066 12061 server_base.cc:1034] running on GCE node
I20260811 12:33:49.280946 12061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:49.282485 12061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:49.283720 12061 hybrid_clock.cc:648] HybridClock initialized: now 1786451629283667 us; error 44 us; skew 500 ppm
I20260811 12:33:49.284123 12061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:33:49.286458 12061 webserver.cc:460] Webserver started at http://127.3.204.126:45803/ using document root <none> and password file <none>
I20260811 12:33:49.287164 12061 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:49.287272 12061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:49.287604 12061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:49.290671 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "56b8130c63e24ac681cfe47492afdaef"
format_stamp: "Formatted at 2026-08-11 12:33:49 on dist-test-slave-lgr5"
server_key: "96540b7a6fe565012dc789a5553456bf"
server_key_iv: "6e327b4082d7018235e593ffd05fe7dc"
server_key_version: "encryptionkey@0"
I20260811 12:33:49.291527 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "56b8130c63e24ac681cfe47492afdaef"
format_stamp: "Formatted at 2026-08-11 12:33:49 on dist-test-slave-lgr5"
server_key: "96540b7a6fe565012dc789a5553456bf"
server_key_iv: "6e327b4082d7018235e593ffd05fe7dc"
server_key_version: "encryptionkey@0"
I20260811 12:33:49.296991 12061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260811 12:33:49.300554 12081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:49.301960 12061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 12:33:49.302134 12061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "56b8130c63e24ac681cfe47492afdaef"
format_stamp: "Formatted at 2026-08-11 12:33:49 on dist-test-slave-lgr5"
server_key: "96540b7a6fe565012dc789a5553456bf"
server_key_iv: "6e327b4082d7018235e593ffd05fe7dc"
server_key_version: "encryptionkey@0"
I20260811 12:33:49.302352 12061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:49.308449 12061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:49.309262 12061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:49.309561 12061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:49.327327 12061 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:38779
I20260811 12:33:49.327368 12169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:38779 every 8 connection(s)
I20260811 12:33:49.328794 12061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:49.332695 12174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:49.336915 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 12061
I20260811 12:33:49.337087 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:49.337765 3889 external_mini_cluster.cc:1428] Setting key bc7e215045cf4f2b07eda38f7f1e7c95
I20260811 12:33:49.344364 12174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef: Bootstrap starting.
I20260811 12:33:49.346357 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:49.339740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.3.204.126:38779)
0811 12:33:49.340763 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:49.340775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:49.340825 (+ 50us) server_negotiation.cc:407] Connection header received
0811 12:33:49.342748 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:49.342775 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:49.343214 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:49.343595 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:49.344706 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:49.344715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:49.344731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:49.344745 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:49.344811 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:49.344827 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:49.345045 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:49.345188 (+ 143us) server_negotiation.cc:299] Negotiation successful
0811 12:33:49.345439 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":371,"threads_started":1}
I20260811 12:33:49.347910 12174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:49.349036 12174 log.cc:824] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:49.351708 12174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef: No bootstrap required, opened a new log
I20260811 12:33:49.359140 12174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b8130c63e24ac681cfe47492afdaef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38779 } }
I20260811 12:33:49.359532 12174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:49.359609 12174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b8130c63e24ac681cfe47492afdaef, State: Initialized, Role: FOLLOWER
I20260811 12:33:49.360205 12174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b8130c63e24ac681cfe47492afdaef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38779 } }
I20260811 12:33:49.360430 12174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:49.360536 12174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:49.360669 12174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:49.363135 12174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b8130c63e24ac681cfe47492afdaef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38779 } }
I20260811 12:33:49.363612 12174 leader_election.cc:302] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56b8130c63e24ac681cfe47492afdaef; no voters:
I20260811 12:33:49.364223 12174 leader_election.cc:288] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:49.364331 12182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:49.365000 12182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [term 1 LEADER]: Becoming Leader. State: Replica: 56b8130c63e24ac681cfe47492afdaef, State: Running, Role: LEADER
I20260811 12:33:49.365582 12182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b8130c63e24ac681cfe47492afdaef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38779 } }
I20260811 12:33:49.366498 12174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:49.368649 12183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56b8130c63e24ac681cfe47492afdaef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b8130c63e24ac681cfe47492afdaef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38779 } } }
I20260811 12:33:49.368646 12184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56b8130c63e24ac681cfe47492afdaef. Latest consensus state: current_term: 1 leader_uuid: "56b8130c63e24ac681cfe47492afdaef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b8130c63e24ac681cfe47492afdaef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38779 } } }
I20260811 12:33:49.369100 12183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:49.369104 12184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:49.370318 12189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:49.375190 12189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:49.386137 12189 catalog_manager.cc:1347] Generated new cluster ID: 23709dbdbf724a4bb0962bd02894ba6b
I20260811 12:33:49.386281 12189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:49.401320 12189 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:49.401747 12189 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:49.410977 12189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef: Generated new TSK 0
I20260811 12:33:49.411857 12189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:33:49.419147 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38779
--builtin_ntp_servers=127.3.204.84:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 12:33:49.758035 12210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:49.758339 12210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:49.758457 12210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:49.765745 12210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:49.765863 12210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:49.766004 12210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:49.775439 12210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:49.776824 12210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:49.778646 12210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:49.792642 12216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:49.792593 12221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:49.792670 12218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:49.793411 12210 server_base.cc:1034] running on GCE node
I20260811 12:33:49.794010 12210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:49.794958 12210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:49.796173 12210 hybrid_clock.cc:648] HybridClock initialized: now 1786451629796121 us; error 34 us; skew 500 ppm
I20260811 12:33:49.796502 12210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:33:49.798725 12210 webserver.cc:460] Webserver started at http://127.3.204.65:38121/ using document root <none> and password file <none>
I20260811 12:33:49.799441 12210 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:49.799576 12210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:49.799912 12210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:49.802959 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "047efbb9122c4fb0bcc8fb2f1290adf1"
format_stamp: "Formatted at 2026-08-11 12:33:49 on dist-test-slave-lgr5"
server_key: "5d9f781a649454dc2812c931bd08ef83"
server_key_iv: "45a5400cf9ca20318bc0dc08e41be0d8"
server_key_version: "encryptionkey@0"
I20260811 12:33:49.803776 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "047efbb9122c4fb0bcc8fb2f1290adf1"
format_stamp: "Formatted at 2026-08-11 12:33:49 on dist-test-slave-lgr5"
server_key: "5d9f781a649454dc2812c931bd08ef83"
server_key_iv: "45a5400cf9ca20318bc0dc08e41be0d8"
server_key_version: "encryptionkey@0"
I20260811 12:33:49.809479 12210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 12:33:49.813026 12231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:49.814378 12210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:33:49.814579 12210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "047efbb9122c4fb0bcc8fb2f1290adf1"
format_stamp: "Formatted at 2026-08-11 12:33:49 on dist-test-slave-lgr5"
server_key: "5d9f781a649454dc2812c931bd08ef83"
server_key_iv: "45a5400cf9ca20318bc0dc08e41be0d8"
server_key_version: "encryptionkey@0"
I20260811 12:33:49.814847 12210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:49.821026 12210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:49.821913 12210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:49.822182 12210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:49.823149 12210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:49.824712 12210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:49.824813 12210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:49.824928 12210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:49.824997 12210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:49.857628 12210 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:33051
I20260811 12:33:49.857702 12407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:33051 every 8 connection(s)
I20260811 12:33:49.859016 12210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:49.863775 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 12210
I20260811 12:33:49.863955 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:49.864619 3889 external_mini_cluster.cc:1428] Setting key 77b552304ebe7ef60238e31b9722c5a9
I20260811 12:33:49.867377 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38779
--builtin_ntp_servers=127.3.204.84:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 12:33:49.867430 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:49.861633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:35311 (local address 127.3.204.126:38779)
0811 12:33:49.862071 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:49.862075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:49.863304 (+ 1229us) server_negotiation.cc:407] Connection header received
0811 12:33:49.865153 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:49.865160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:49.865227 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:49.865327 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:49.866681 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:49.866688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:49.866692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:49.866710 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:49.866753 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:49.866766 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:49.866872 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:49.867069 (+ 197us) server_negotiation.cc:299] Negotiation successful
0811 12:33:49.867137 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":232,"threads_started":1}
I20260811 12:33:49.868481 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:49.862184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:38779 (local address 127.3.204.65:35311)
0811 12:33:49.863128 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:49.863169 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:49.864879 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:49.865595 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:49.865604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:49.865699 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:49.866349 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:33:49.866351 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:49.866393 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:33:49.866485 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:49.866857 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:49.866868 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:49.866883 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 12:33:49.867141 (+ 258us) client_negotiation.cc:240] Negotiation successful
0811 12:33:49.867412 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":270,"threads_started":1}
I20260811 12:33:49.870103 12409 heartbeater.cc:347] Connected to a master server at 127.3.204.126:38779
I20260811 12:33:49.870368 12409 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:49.871089 12409 heartbeater.cc:511] Master 127.3.204.126:38779 requested a full tablet report, sending...
I20260811 12:33:49.873703 12105 ts_manager.cc:194] Registered new tserver with Master: 047efbb9122c4fb0bcc8fb2f1290adf1 (127.3.204.65:33051)
W20260811 12:33:50.219705 12414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:50.220070 12414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:50.220237 12414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:50.227540 12414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:50.227667 12414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:50.227813 12414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:50.237249 12414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:50.238590 12414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:50.240463 12414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:50.253537 12422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:50.253516 12421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:50.254070 12425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:50.254652 12414 server_base.cc:1034] running on GCE node
I20260811 12:33:50.255306 12414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:50.256337 12414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:50.257560 12414 hybrid_clock.cc:648] HybridClock initialized: now 1786451630257508 us; error 61 us; skew 500 ppm
I20260811 12:33:50.257885 12414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:33:50.260214 12414 webserver.cc:460] Webserver started at http://127.3.204.66:43069/ using document root <none> and password file <none>
I20260811 12:33:50.260951 12414 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:50.261065 12414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:50.261386 12414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:50.264595 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "0328f76141ca4d9eb6fca61ce41807b5"
format_stamp: "Formatted at 2026-08-11 12:33:50 on dist-test-slave-lgr5"
server_key: "dcae09233e52f60003ece16441051065"
server_key_iv: "0d8b65af8a2c0af25a4868ba156fc012"
server_key_version: "encryptionkey@0"
I20260811 12:33:50.265437 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "0328f76141ca4d9eb6fca61ce41807b5"
format_stamp: "Formatted at 2026-08-11 12:33:50 on dist-test-slave-lgr5"
server_key: "dcae09233e52f60003ece16441051065"
server_key_iv: "0d8b65af8a2c0af25a4868ba156fc012"
server_key_version: "encryptionkey@0"
I20260811 12:33:50.271112 12414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:50.274752 12434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:50.276253 12414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:50.276392 12414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "0328f76141ca4d9eb6fca61ce41807b5"
format_stamp: "Formatted at 2026-08-11 12:33:50 on dist-test-slave-lgr5"
server_key: "dcae09233e52f60003ece16441051065"
server_key_iv: "0d8b65af8a2c0af25a4868ba156fc012"
server_key_version: "encryptionkey@0"
I20260811 12:33:50.276602 12414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:50.282459 12414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:50.283286 12414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:50.283531 12414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:50.284528 12414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:50.285971 12414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:50.286060 12414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:50.286145 12414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:50.286198 12414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:50.316666 12414 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:44133
I20260811 12:33:50.316715 12611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:44133 every 8 connection(s)
I20260811 12:33:50.318037 12414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:50.325203 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 12414
I20260811 12:33:50.325356 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:50.325798 3889 external_mini_cluster.cc:1428] Setting key f68423091478dc2a29c6cb4e6b2f3a4f
I20260811 12:33:50.326363 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.320806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:34821 (local address 127.3.204.126:38779)
0811 12:33:50.320980 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:50.320985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:50.322355 (+ 1370us) server_negotiation.cc:407] Connection header received
0811 12:33:50.324045 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:50.324049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:50.324150 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:50.324242 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:50.325561 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:50.325568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:50.325573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:50.325590 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:50.325625 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:50.325638 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:50.325748 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:50.325991 (+ 243us) server_negotiation.cc:299] Negotiation successful
0811 12:33:50.326064 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 12:33:50.327306 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.321295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:38779 (local address 127.3.204.66:34821)
0811 12:33:50.322214 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:50.322260 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:50.323809 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:50.324447 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:50.324455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:50.324545 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:50.325215 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:33:50.325218 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:50.325257 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:33:50.325380 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:50.325728 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:50.325738 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:50.325746 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 12:33:50.326018 (+ 272us) client_negotiation.cc:240] Negotiation successful
0811 12:33:50.326223 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":284,"threads_started":1}
I20260811 12:33:50.328872 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38779
--builtin_ntp_servers=127.3.204.84:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 12:33:50.337443 12612 heartbeater.cc:347] Connected to a master server at 127.3.204.126:38779
I20260811 12:33:50.337808 12612 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:50.338496 12612 heartbeater.cc:511] Master 127.3.204.126:38779 requested a full tablet report, sending...
I20260811 12:33:50.340488 12105 ts_manager.cc:194] Registered new tserver with Master: 0328f76141ca4d9eb6fca61ce41807b5 (127.3.204.66:44133)
W20260811 12:33:50.664137 12616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:50.664436 12616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:50.664529 12616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:50.671813 12616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:50.671931 12616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:50.672050 12616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:50.682106 12616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:50.683422 12616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:50.685304 12616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:50.698051 12633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:50.698078 12637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:50.698140 12635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:50.701591 12616 server_base.cc:1034] running on GCE node
I20260811 12:33:50.702212 12616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:50.703173 12616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:50.704370 12616 hybrid_clock.cc:648] HybridClock initialized: now 1786451630704323 us; error 40 us; skew 500 ppm
I20260811 12:33:50.704689 12616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:33:50.707072 12616 webserver.cc:460] Webserver started at http://127.3.204.67:37715/ using document root <none> and password file <none>
I20260811 12:33:50.707782 12616 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:50.707875 12616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:50.708279 12616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:50.711350 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "b6ebf6974758444dbbe8e5c947c68171"
format_stamp: "Formatted at 2026-08-11 12:33:50 on dist-test-slave-lgr5"
server_key: "105e1367126545f88fba1e28e2fbde4d"
server_key_iv: "3bac88f555fbec0e01d4e89bd7af30ab"
server_key_version: "encryptionkey@0"
I20260811 12:33:50.712186 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "b6ebf6974758444dbbe8e5c947c68171"
format_stamp: "Formatted at 2026-08-11 12:33:50 on dist-test-slave-lgr5"
server_key: "105e1367126545f88fba1e28e2fbde4d"
server_key_iv: "3bac88f555fbec0e01d4e89bd7af30ab"
server_key_version: "encryptionkey@0"
I20260811 12:33:50.717517 12616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260811 12:33:50.721103 12645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:50.722429 12616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:33:50.722589 12616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "b6ebf6974758444dbbe8e5c947c68171"
format_stamp: "Formatted at 2026-08-11 12:33:50 on dist-test-slave-lgr5"
server_key: "105e1367126545f88fba1e28e2fbde4d"
server_key_iv: "3bac88f555fbec0e01d4e89bd7af30ab"
server_key_version: "encryptionkey@0"
I20260811 12:33:50.722805 12616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:50.728943 12616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:50.729852 12616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:50.730132 12616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:50.731115 12616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:50.732698 12616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:50.732793 12616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:50.732885 12616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:50.732954 12616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:50.763799 12616 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:43449
I20260811 12:33:50.763850 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:43449 every 8 connection(s)
I20260811 12:33:50.765239 12616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:50.773460 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.767817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:45695 (local address 127.3.204.126:38779)
0811 12:33:50.767987 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:50.767991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:50.769470 (+ 1479us) server_negotiation.cc:407] Connection header received
0811 12:33:50.771203 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:50.771209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:50.771277 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:50.771370 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:50.772716 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:50.772722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:50.772727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:50.772749 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:50.772785 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:50.772797 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:50.772920 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:50.773102 (+ 182us) server_negotiation.cc:299] Negotiation successful
0811 12:33:50.773177 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 12:33:50.774488 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.768396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:38779 (local address 127.3.204.67:45695)
0811 12:33:50.769320 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:50.769357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:50.770920 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:50.771613 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:50.771622 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:50.771714 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:50.772419 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:33:50.772421 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:50.772460 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:33:50.772545 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:50.772893 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:50.772904 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:50.772912 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 12:33:50.773173 (+ 261us) client_negotiation.cc:240] Negotiation successful
0811 12:33:50.773443 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":266,"threads_started":1}
I20260811 12:33:50.774487 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 12616
I20260811 12:33:50.774634 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:50.775206 3889 external_mini_cluster.cc:1428] Setting key 3a74394d384f6fd2a5903402c8d1f467
I20260811 12:33:50.775846 12827 heartbeater.cc:347] Connected to a master server at 127.3.204.126:38779
I20260811 12:33:50.776181 12827 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:50.776878 12827 heartbeater.cc:511] Master 127.3.204.126:38779 requested a full tablet report, sending...
I20260811 12:33:50.778306 12105 ts_manager.cc:194] Registered new tserver with Master: b6ebf6974758444dbbe8e5c947c68171 (127.3.204.67:43449)
I20260811 12:33:50.790518 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:33:50.797804 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.795511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.3.204.126:38779)
0811 12:33:50.795682 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:50.795687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:50.796167 (+ 480us) server_negotiation.cc:407] Connection header received
0811 12:33:50.796327 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:50.796331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:50.796385 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:50.796474 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:50.797287 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:50.797291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:50.797294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:50.797310 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:50.797351 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:50.797364 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:50.797468 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:50.797530 (+ 62us) server_negotiation.cc:299] Negotiation successful
0811 12:33:50.797578 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 12:33:50.801648 12105 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:33:50.808683 12105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:33:50.828493 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.822589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.3.204.66:44133)
0811 12:33:50.823266 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:50.823275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:50.824618 (+ 1343us) server_negotiation.cc:407] Connection header received
0811 12:33:50.824793 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:50.824815 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:50.825072 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:50.825230 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:50.827425 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:50.827433 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:50.827447 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:50.827478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:50.827543 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:50.827565 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:50.827737 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:50.828023 (+ 286us) server_negotiation.cc:299] Negotiation successful
0811 12:33:50.828168 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":362,"threads_started":1}
I20260811 12:33:50.829010 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.822898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.0.0.1:55348)
0811 12:33:50.824025 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:50.824044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:50.824165 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:50.825288 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:50.825296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:50.825396 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:50.826167 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:33:50.826170 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:50.826194 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:33:50.826276 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:50.827357 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:50.827370 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:50.827379 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 12:33:50.827559 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 12:33:50.827673 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":859,"threads_started":1}
I20260811 12:33:50.829120 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.821114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43449 (local address 127.0.0.1:42118)
0811 12:33:50.822507 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:50.822550 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:50.822677 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:50.826167 (+ 3490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:50.826171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:50.826197 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:50.826952 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:33:50.826956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:50.826972 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:33:50.827031 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:50.828272 (+ 1241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:50.828278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:50.828283 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 12:33:50.828599 (+ 316us) client_negotiation.cc:240] Negotiation successful
0811 12:33:50.828661 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":970,"threads_started":1}
I20260811 12:33:50.829588 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.821305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.3.204.67:43449)
0811 12:33:50.825644 (+ 4339us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:50.825652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:50.825687 (+ 35us) server_negotiation.cc:407] Connection header received
0811 12:33:50.825756 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:50.825763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:50.825969 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:50.826655 (+ 686us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:50.827897 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:50.827906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:50.827919 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:50.827939 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:50.828003 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:50.828029 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:50.828497 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:50.829115 (+ 618us) server_negotiation.cc:299] Negotiation successful
0811 12:33:50.829225 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4157,"thread_start_us":3981,"threads_started":1}
I20260811 12:33:50.829773 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.821810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:44133 (local address 127.0.0.1:46780)
0811 12:33:50.823744 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:50.823764 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:50.823887 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:50.826341 (+ 2454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:50.826346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:50.826367 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:50.827220 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:33:50.827223 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:50.827239 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:33:50.827276 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:50.827685 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:50.827689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:50.827694 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 12:33:50.828760 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0811 12:33:50.828819 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1773,"thread_start_us":1620,"threads_started":1}
I20260811 12:33:50.830027 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:50.823487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.3.204.65:33051)
0811 12:33:50.824673 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:50.824683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:50.824713 (+ 30us) server_negotiation.cc:407] Connection header received
0811 12:33:50.824791 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:50.824800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:50.825062 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:50.825812 (+ 750us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:50.827132 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:50.827138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:50.827149 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:50.827162 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:50.827213 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:50.827234 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:50.829514 (+ 2280us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:50.829604 (+ 90us) server_negotiation.cc:299] Negotiation successful
0811 12:33:50.829700 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":858,"threads_started":1}
I20260811 12:33:50.833354 12312 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1ffe88bda84f298d3614d2898e923a (DEFAULT_TABLE table=test-table [id=a3c4149cc43d4c55bbce17e71d94d817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:50.833375 12508 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1ffe88bda84f298d3614d2898e923a (DEFAULT_TABLE table=test-table [id=a3c4149cc43d4c55bbce17e71d94d817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:50.833354 12727 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1ffe88bda84f298d3614d2898e923a (DEFAULT_TABLE table=test-table [id=a3c4149cc43d4c55bbce17e71d94d817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:33:50.834816 12727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1ffe88bda84f298d3614d2898e923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:50.834815 12312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1ffe88bda84f298d3614d2898e923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:50.834831 12508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1ffe88bda84f298d3614d2898e923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:50.846638 12850 tablet_bootstrap.cc:492] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Bootstrap starting.
I20260811 12:33:50.849888 12853 tablet_bootstrap.cc:492] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: Bootstrap starting.
I20260811 12:33:50.851107 12850 tablet_bootstrap.cc:654] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:50.852497 12850 log.cc:824] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:50.853693 12853 tablet_bootstrap.cc:654] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:50.854063 12854 tablet_bootstrap.cc:492] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Bootstrap starting.
I20260811 12:33:50.855693 12853 log.cc:824] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:50.855906 12850 tablet_bootstrap.cc:492] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: No bootstrap required, opened a new log
I20260811 12:33:50.856213 12850 ts_tablet_manager.cc:1397] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260811 12:33:50.857573 12854 tablet_bootstrap.cc:654] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:50.858834 12854 log.cc:824] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:50.860355 12853 tablet_bootstrap.cc:492] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: No bootstrap required, opened a new log
I20260811 12:33:50.860743 12853 ts_tablet_manager.cc:1397] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260811 12:33:50.866178 12850 raft_consensus.cc:348] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:50.866570 12850 raft_consensus.cc:374] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:50.866695 12850 raft_consensus.cc:729] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0328f76141ca4d9eb6fca61ce41807b5, State: Initialized, Role: FOLLOWER
I20260811 12:33:50.867337 12850 consensus_queue.cc:260] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:50.868418 12612 heartbeater.cc:503] Master 127.3.204.126:38779 was elected leader, sending a full tablet report...
I20260811 12:33:50.870273 12850 ts_tablet_manager.cc:1428] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260811 12:33:50.873178 12854 tablet_bootstrap.cc:492] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: No bootstrap required, opened a new log
I20260811 12:33:50.873519 12854 ts_tablet_manager.cc:1397] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20260811 12:33:50.873508 12853 raft_consensus.cc:348] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:50.873967 12853 raft_consensus.cc:374] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:50.874079 12853 raft_consensus.cc:729] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047efbb9122c4fb0bcc8fb2f1290adf1, State: Initialized, Role: FOLLOWER
I20260811 12:33:50.874738 12853 consensus_queue.cc:260] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:50.876792 12409 heartbeater.cc:503] Master 127.3.204.126:38779 was elected leader, sending a full tablet report...
I20260811 12:33:50.877344 12853 ts_tablet_manager.cc:1428] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260811 12:33:50.882962 12854 raft_consensus.cc:348] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:50.883319 12854 raft_consensus.cc:374] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:50.883391 12854 raft_consensus.cc:729] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ebf6974758444dbbe8e5c947c68171, State: Initialized, Role: FOLLOWER
I20260811 12:33:50.883898 12854 consensus_queue.cc:260] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:50.884886 12827 heartbeater.cc:503] Master 127.3.204.126:38779 was elected leader, sending a full tablet report...
I20260811 12:33:50.885329 12854 ts_tablet_manager.cc:1428] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
W20260811 12:33:51.016830 12830 tablet.cc:2369] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:33:51.070178 12613 tablet.cc:2369] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:33:51.111501 12410 tablet.cc:2369] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:33:51.134332 12860 raft_consensus.cc:482] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:33:51.134662 12860 raft_consensus.cc:504] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:51.136539 12860 leader_election.cc:288] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ebf6974758444dbbe8e5c947c68171 (127.3.204.67:43449), 047efbb9122c4fb0bcc8fb2f1290adf1 (127.3.204.65:33051)
I20260811 12:33:51.139986 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.137132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:37231 (local address 127.3.204.67:43449)
0811 12:33:51.137324 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:51.137329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:51.137780 (+ 451us) server_negotiation.cc:407] Connection header received
0811 12:33:51.137963 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:51.137969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:51.138040 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:51.138170 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:51.139375 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:51.139381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:51.139387 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:51.139406 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:51.139446 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:51.139463 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:51.139591 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:51.139687 (+ 96us) server_negotiation.cc:299] Negotiation successful
0811 12:33:51.139755 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:33:51.140328 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.136862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43449 (local address 127.3.204.66:37231)
0811 12:33:51.137632 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.137653 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:51.137805 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:51.138228 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:51.138233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:51.138264 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:51.139138 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:33:51.139142 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:51.139159 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:33:51.139209 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:51.139562 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:51.139567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:51.139571 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 12:33:51.139720 (+ 149us) client_negotiation.cc:240] Negotiation successful
0811 12:33:51.139781 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":336,"threads_started":1}
I20260811 12:33:51.140622 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.137884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:53287 (local address 127.3.204.65:33051)
0811 12:33:51.138074 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:51.138079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:51.138588 (+ 509us) server_negotiation.cc:407] Connection header received
0811 12:33:51.138703 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:51.138709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:51.138768 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:51.138862 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:51.139872 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:51.139879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:51.139885 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:51.139908 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:51.139951 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:51.139969 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:51.140124 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:51.140249 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 12:33:51.140317 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 12:33:51.140761 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.137704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.3.204.66:53287)
0811 12:33:51.138465 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.138480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:51.138593 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:51.138905 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:51.138910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:51.138932 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:51.139637 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:33:51.139640 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:51.139657 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:33:51.139705 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:51.140114 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:51.140118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:51.140123 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 12:33:51.140293 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 12:33:51.140360 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":370,"threads_started":1}
I20260811 12:33:51.141511 12752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1ffe88bda84f298d3614d2898e923a" candidate_uuid: "0328f76141ca4d9eb6fca61ce41807b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ebf6974758444dbbe8e5c947c68171" is_pre_election: true
I20260811 12:33:51.141765 12341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1ffe88bda84f298d3614d2898e923a" candidate_uuid: "0328f76141ca4d9eb6fca61ce41807b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" is_pre_election: true
I20260811 12:33:51.142053 12752 raft_consensus.cc:2454] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0328f76141ca4d9eb6fca61ce41807b5 in term 0.
I20260811 12:33:51.142338 12341 raft_consensus.cc:2454] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0328f76141ca4d9eb6fca61ce41807b5 in term 0.
I20260811 12:33:51.142750 12437 leader_election.cc:302] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0328f76141ca4d9eb6fca61ce41807b5, b6ebf6974758444dbbe8e5c947c68171; no voters:
I20260811 12:33:51.143177 12860 raft_consensus.cc:2781] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:33:51.143330 12860 raft_consensus.cc:482] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:33:51.143428 12860 raft_consensus.cc:3037] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:51.145685 12860 raft_consensus.cc:504] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:51.146291 12860 leader_election.cc:288] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [CANDIDATE]: Term 1 election: Requested vote from peers b6ebf6974758444dbbe8e5c947c68171 (127.3.204.67:43449), 047efbb9122c4fb0bcc8fb2f1290adf1 (127.3.204.65:33051)
I20260811 12:33:51.146852 12752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1ffe88bda84f298d3614d2898e923a" candidate_uuid: "0328f76141ca4d9eb6fca61ce41807b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ebf6974758444dbbe8e5c947c68171"
I20260811 12:33:51.146862 12341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1ffe88bda84f298d3614d2898e923a" candidate_uuid: "0328f76141ca4d9eb6fca61ce41807b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1"
I20260811 12:33:51.147058 12752 raft_consensus.cc:3037] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:51.147078 12341 raft_consensus.cc:3037] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:51.149391 12752 raft_consensus.cc:2454] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0328f76141ca4d9eb6fca61ce41807b5 in term 1.
I20260811 12:33:51.149415 12341 raft_consensus.cc:2454] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0328f76141ca4d9eb6fca61ce41807b5 in term 1.
I20260811 12:33:51.149832 12437 leader_election.cc:302] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0328f76141ca4d9eb6fca61ce41807b5, b6ebf6974758444dbbe8e5c947c68171; no voters:
I20260811 12:33:51.150111 12860 raft_consensus.cc:2781] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:51.150568 12860 raft_consensus.cc:686] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 1 LEADER]: Becoming Leader. State: Replica: 0328f76141ca4d9eb6fca61ce41807b5, State: Running, Role: LEADER
I20260811 12:33:51.151295 12860 consensus_queue.cc:237] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } }
I20260811 12:33:51.156708 12105 catalog_manager.cc:5495] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0328f76141ca4d9eb6fca61ce41807b5 (127.3.204.66). New cstate: current_term: 1 leader_uuid: "0328f76141ca4d9eb6fca61ce41807b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0328f76141ca4d9eb6fca61ce41807b5" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 44133 } health_report { overall_health: HEALTHY } } }
I20260811 12:33:51.200150 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.198209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.3.204.66:44133)
0811 12:33:51.198427 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:51.198432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:51.198450 (+ 18us) server_negotiation.cc:407] Connection header received
0811 12:33:51.198659 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:51.198663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:51.198755 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:51.198843 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:51.199614 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:51.199618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:51.199621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:51.199636 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:51.199669 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:51.199680 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:51.199781 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:51.199816 (+ 35us) server_negotiation.cc:299] Negotiation successful
0811 12:33:51.199861 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:33:51.209147 12752 raft_consensus.cc:1261] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 1 FOLLOWER]: Refusing update from remote peer 0328f76141ca4d9eb6fca61ce41807b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:51.209147 12341 raft_consensus.cc:1261] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 1 FOLLOWER]: Refusing update from remote peer 0328f76141ca4d9eb6fca61ce41807b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:33:51.210296 12866 consensus_queue.cc:1035] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047efbb9122c4fb0bcc8fb2f1290adf1" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:51.210681 12860 consensus_queue.cc:1035] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ebf6974758444dbbe8e5c947c68171" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:33:51.224067 12871 mvcc.cc:205] Tried to move back new op lower bound from 7317305881420345344 to 7317305881200017408. Current Snapshot: MvccSnapshot[applied={T|T < 7317305881420345344}]
I20260811 12:33:51.236553 12104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:51.236912 12104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:33:51.241024 12104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56b8130c63e24ac681cfe47492afdaef: Sending DeleteTablet for 3 replicas of tablet 9d1ffe88bda84f298d3614d2898e923a
I20260811 12:33:51.242463 12312 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1ffe88bda84f298d3614d2898e923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:51 UTC) from {username='slave'} at 127.0.0.1:55348
I20260811 12:33:51.242470 12508 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1ffe88bda84f298d3614d2898e923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:51 UTC) from {username='slave'} at 127.0.0.1:46780
I20260811 12:33:51.242888 12727 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1ffe88bda84f298d3614d2898e923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:33:51 UTC) from {username='slave'} at 127.0.0.1:42118
I20260811 12:33:51.244915 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 12210
I20260811 12:33:51.255226 12883 tablet_replica.cc:330] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: stopping tablet replica
I20260811 12:33:51.255764 12883 raft_consensus.cc:2229] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:51.256403 12883 raft_consensus.cc:2258] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:51.257313 12886 tablet_replica.cc:330] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: stopping tablet replica
I20260811 12:33:51.257766 12886 raft_consensus.cc:2229] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:33:51.258343 12886 raft_consensus.cc:2258] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:51.259589 12883 ts_tablet_manager.cc:1905] T 9d1ffe88bda84f298d3614d2898e923a P 047efbb9122c4fb0bcc8fb2f1290adf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:51.262094 12886 ts_tablet_manager.cc:1905] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:51.262585 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.246512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.3.204.65:33051)
0811 12:33:51.246714 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:51.246721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:51.259273 (+ 12552us) server_negotiation.cc:407] Connection header received
0811 12:33:51.260071 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:51.260125 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:51.260231 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:51.260358 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:51.261516 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:51.261526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:51.261531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:51.261552 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:51.261593 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:51.261613 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:51.261747 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:51.262236 (+ 489us) server_negotiation.cc:299] Negotiation successful
0811 12:33:51.262306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 12:33:51.262946 12885 tablet_replica.cc:330] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: stopping tablet replica
I20260811 12:33:51.263517 12397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:51.263510 12885 raft_consensus.cc:2229] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:33:51.264315 12885 raft_consensus.cc:2258] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:33:51.267086 12885 ts_tablet_manager.cc:1905] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:33:51.271428 12886 ts_tablet_manager.cc:1918] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:33:51.271636 12886 log.cc:1196] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/9d1ffe88bda84f298d3614d2898e923a
I20260811 12:33:51.272215 12886 ts_tablet_manager.cc:1939] T 9d1ffe88bda84f298d3614d2898e923a P b6ebf6974758444dbbe8e5c947c68171: Deleting consensus metadata
I20260811 12:33:51.274360 12086 catalog_manager.cc:4841] TS b6ebf6974758444dbbe8e5c947c68171 (127.3.204.67:43449): tablet 9d1ffe88bda84f298d3614d2898e923a (table test-table [id=a3c4149cc43d4c55bbce17e71d94d817]) successfully deleted
I20260811 12:33:51.278201 12885 ts_tablet_manager.cc:1918] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:33:51.278451 12885 log.cc:1196] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/9d1ffe88bda84f298d3614d2898e923a
I20260811 12:33:51.278990 12885 ts_tablet_manager.cc:1939] T 9d1ffe88bda84f298d3614d2898e923a P 0328f76141ca4d9eb6fca61ce41807b5: Deleting consensus metadata
I20260811 12:33:51.280723 12088 catalog_manager.cc:4841] TS 0328f76141ca4d9eb6fca61ce41807b5 (127.3.204.66:44133): tablet 9d1ffe88bda84f298d3614d2898e923a (table test-table [id=a3c4149cc43d4c55bbce17e71d94d817]) successfully deleted
I20260811 12:33:51.409955 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 12210
W20260811 12:33:51.441386 12086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.65:33051 (error 108)
I20260811 12:33:51.441694 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 12414
I20260811 12:33:51.443764 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.442953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.0.0.1:55370)
0811 12:33:51.443402 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.443555 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":211,"threads_started":1}
W20260811 12:33:51.444052 12086 catalog_manager.cc:4584] TS 047efbb9122c4fb0bcc8fb2f1290adf1 (127.3.204.65:33051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d1ffe88bda84f298d3614d2898e923a: Network error: Client connection negotiation failed: client connection to 127.3.204.65:33051: connect: Connection refused (error 111)
I20260811 12:33:51.445116 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.442622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.3.204.66:44133)
0811 12:33:51.442804 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:51.442810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:51.442831 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:33:51.443047 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:51.443054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:51.443122 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:51.443228 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:51.444436 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:51.444442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:51.444448 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:51.444468 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:51.444511 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:51.444527 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:51.444667 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:51.444746 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 12:33:51.444807 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 12:33:51.445851 12602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:51.497870 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.497284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.0.0.1:55382)
0811 12:33:51.497535 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.497682 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 12:33:51.498982 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.498578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.0.0.1:55390)
0811 12:33:51.498723 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.498807 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 12:33:51.558389 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 12414
I20260811 12:33:51.564934 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.564445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.0.0.1:55392)
0811 12:33:51.564698 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.564784 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260811 12:33:51.567847 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.565856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.0.0.1:55398)
0811 12:33:51.567516 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.567624 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1525,"threads_started":1}
I20260811 12:33:51.587755 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 12616
I20260811 12:33:51.590737 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.588609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.3.204.67:43449)
0811 12:33:51.588814 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:51.588819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:51.588842 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:33:51.588988 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:51.588992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:51.589097 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:51.589188 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:51.590094 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:51.590100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:51.590104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:51.590120 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:51.590155 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:51.590167 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:51.590284 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:51.590377 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 12:33:51.590446 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 12:33:51.591377 12819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:51.649727 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.649116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.0.0.1:55414)
0811 12:33:51.649400 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.649529 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260811 12:33:51.650983 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:51.650516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33051 (local address 127.0.0.1:55422)
0811 12:33:51.650762 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:51.650862 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260811 12:33:51.679807 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 12616
I20260811 12:33:51.703810 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 12061
I20260811 12:33:51.705098 12156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:51.811894 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 12061
2026-08-11T12:33:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2983 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260811 12:33:51.860595 3889 mini_oidc.cc:152] Starting JWKS server
I20260811 12:33:51.865056 3889 webserver.cc:460] Webserver started at https://127.0.0.1:34677/ using document root <none> and password file <none>
I20260811 12:33:51.865407 3889 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 12:33:51.865999 3889 webserver.cc:460] Webserver started at http://127.0.0.1:43039/ using document root <none> and password file <none>
2026-08-11T12:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:51Z Disabled control of system clock
I20260811 12:33:51.896567 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:45453
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:45453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34677/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/testchainca.cert with env {}
W20260811 12:33:52.235674 12906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:52.236034 12906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:52.236197 12906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:52.243613 12906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:52.243744 12906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:52.243816 12906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:52.243893 12906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:52.243954 12906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 12:33:52.244050 12906 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 12:33:52.244123 12906 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34677/jwks.json
I20260811 12:33:52.254262 12906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:45453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34677/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:52.255676 12906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:52.257054 12906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:52.269217 12913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:52.269186 12914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:52.269186 12916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:52.270226 12906 server_base.cc:1034] running on GCE node
I20260811 12:33:52.271016 12906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:52.272550 12906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:52.273758 12906 hybrid_clock.cc:648] HybridClock initialized: now 1786451632273709 us; error 44 us; skew 500 ppm
I20260811 12:33:52.274128 12906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:33:52.282379 12906 webserver.cc:460] Webserver started at http://127.3.204.126:35813/ using document root <none> and password file <none>
I20260811 12:33:52.283255 12906 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:52.283383 12906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:52.283706 12906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:52.288905 12906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "ea8fb9b2235f4c94b41f0371f9839b61"
format_stamp: "Formatted at 2026-08-11 12:33:52 on dist-test-slave-lgr5"
server_key: "fe5c4e30310aea5ff8c6cfed31e45c32"
server_key_iv: "042912e4cb26fde18f7a5de3e5107253"
server_key_version: "encryptionkey@0"
I20260811 12:33:52.289739 12906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "ea8fb9b2235f4c94b41f0371f9839b61"
format_stamp: "Formatted at 2026-08-11 12:33:52 on dist-test-slave-lgr5"
server_key: "fe5c4e30310aea5ff8c6cfed31e45c32"
server_key_iv: "042912e4cb26fde18f7a5de3e5107253"
server_key_version: "encryptionkey@0"
I20260811 12:33:52.295125 12906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:52.298864 12926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:52.300328 12906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:52.300514 12906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "ea8fb9b2235f4c94b41f0371f9839b61"
format_stamp: "Formatted at 2026-08-11 12:33:52 on dist-test-slave-lgr5"
server_key: "fe5c4e30310aea5ff8c6cfed31e45c32"
server_key_iv: "042912e4cb26fde18f7a5de3e5107253"
server_key_version: "encryptionkey@0"
I20260811 12:33:52.300742 12906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:52.327481 12906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:52.328474 12906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:52.328794 12906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:52.347229 12906 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:45453
I20260811 12:33:52.347250 13007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:45453 every 8 connection(s)
I20260811 12:33:52.348745 12906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:52.352751 13011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:52.358024 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 12906
I20260811 12:33:52.358264 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:52.359064 3889 external_mini_cluster.cc:1428] Setting key d476641a1b20c075d2ece5c71bce7618
I20260811 12:33:52.364847 13011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61: Bootstrap starting.
I20260811 12:33:52.368532 13011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:52.369701 13011 log.cc:824] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:52.372830 13011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61: No bootstrap required, opened a new log
I20260811 12:33:52.373492 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:52.361218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.3.204.126:45453)
0811 12:33:52.362334 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:52.362346 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:52.362393 (+ 47us) server_negotiation.cc:407] Connection header received
0811 12:33:52.364179 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:52.364205 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:52.364691 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:52.365078 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:52.365753 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:52.367211 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:52.369036 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:52.369717 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:52.369878 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:52.371104 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:52.371112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:52.371127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:52.371655 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:52.371738 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:52.371760 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:52.371846 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:52.372128 (+ 282us) server_negotiation.cc:299] Negotiation successful
0811 12:33:52.372467 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":375,"threads_started":1}
I20260811 12:33:52.377351 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:33:52.381633 13011 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8fb9b2235f4c94b41f0371f9839b61" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45453 } }
I20260811 12:33:52.381995 13011 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:52.382069 13011 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8fb9b2235f4c94b41f0371f9839b61, State: Initialized, Role: FOLLOWER
I20260811 12:33:52.382560 13011 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8fb9b2235f4c94b41f0371f9839b61" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45453 } }
I20260811 12:33:52.382757 13011 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:52.382853 13011 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:52.382987 13011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:52.385185 13011 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8fb9b2235f4c94b41f0371f9839b61" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45453 } }
I20260811 12:33:52.385660 13011 leader_election.cc:302] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea8fb9b2235f4c94b41f0371f9839b61; no voters:
I20260811 12:33:52.386210 13011 leader_election.cc:288] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:52.386349 13016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:52.387390 13016 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [term 1 LEADER]: Becoming Leader. State: Replica: ea8fb9b2235f4c94b41f0371f9839b61, State: Running, Role: LEADER
I20260811 12:33:52.388072 13016 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8fb9b2235f4c94b41f0371f9839b61" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45453 } }
I20260811 12:33:52.388499 13011 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:52.391053 13017 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea8fb9b2235f4c94b41f0371f9839b61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8fb9b2235f4c94b41f0371f9839b61" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45453 } } }
I20260811 12:33:52.391088 13018 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea8fb9b2235f4c94b41f0371f9839b61. Latest consensus state: current_term: 1 leader_uuid: "ea8fb9b2235f4c94b41f0371f9839b61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8fb9b2235f4c94b41f0371f9839b61" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45453 } } }
I20260811 12:33:52.391476 13017 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:52.391476 13018 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:52.393136 13024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:52.398752 13024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:52.408973 13024 catalog_manager.cc:1347] Generated new cluster ID: b3aa8816d59947e69801b59105e42de9
I20260811 12:33:52.409152 13024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:52.416592 13024 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:52.418416 13024 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:52.431718 13024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea8fb9b2235f4c94b41f0371f9839b61: Generated new TSK 0
I20260811 12:33:52.432957 13024 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 12:33:52.479594 13054 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260811 12:33:52.479857 13054 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 12:33:52.482156 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:52.450817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.3.204.126:45453)
0811 12:33:52.451001 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:52.451008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:52.451477 (+ 469us) server_negotiation.cc:407] Connection header received
0811 12:33:52.451892 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:52.451896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:52.451978 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:52.452124 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0811 12:33:52.452688 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:52.453428 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:52.454864 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:52.455355 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:52.455467 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:52.456032 (+ 565us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 12:33:52.481353 (+ 25321us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34677/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 12:33:52.481808 (+ 455us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34677/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260811 12:33:52.482322 13014 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34677/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260811 12:33:52.484236 13050 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:33:52.450759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45453 (local address 127.0.0.1:42750)
0811 12:33:52.451362 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:52.451378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:52.451557 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:52.452179 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:52.452184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:52.452197 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0811 12:33:52.452516 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:52.452525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:52.453574 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:52.453578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:52.454709 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:52.454721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:52.455539 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:52.455546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:52.455685 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:52.455750 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 12:33:52.481740 (+ 25990us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34677/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 12:33:52.482146 (+ 406us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.204.126:45453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34677/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":310,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.204.126:45453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34677/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260811 12:33:52.486075 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 12906
I20260811 12:33:52.487388 13000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:52.581640 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 12906
2026-08-11T12:33:52Z chronyd exiting
I20260811 12:33:52.637308 3889 test_util.cc:175] -----------------------------------------------
I20260811 12:33:52.637454 3889 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786451585252647-3889-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (782 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260811 12:33:52.641194 3889 mini_oidc.cc:152] Starting JWKS server
I20260811 12:33:52.643033 3889 webserver.cc:460] Webserver started at https://127.0.0.1:46789/ using document root <none> and password file <none>
I20260811 12:33:52.643232 3889 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 12:33:52.643882 3889 webserver.cc:460] Webserver started at http://127.0.0.1:39315/ using document root <none> and password file <none>
2026-08-11T12:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:52Z Disabled control of system clock
I20260811 12:33:52.674742 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:43803
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:35671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:43803
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46789/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/testchainca.cert with env {}
W20260811 12:33:53.010329 13069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:53.010627 13069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:53.010715 13069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:53.018028 13069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:53.018153 13069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:53.018191 13069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:53.018244 13069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:53.018287 13069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 12:33:53.018348 13069 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 12:33:53.018386 13069 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46789/jwks.json
I20260811 12:33:53.028518 13069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:35671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:43803
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46789/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:53.029842 13069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:53.031224 13069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:53.043197 13074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:53.043208 13081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:53.043251 13078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:53.044034 13069 server_base.cc:1034] running on GCE node
I20260811 12:33:53.044871 13069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:53.046310 13069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:53.047513 13069 hybrid_clock.cc:648] HybridClock initialized: now 1786451633047466 us; error 38 us; skew 500 ppm
I20260811 12:33:53.047863 13069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:33:53.050140 13069 webserver.cc:460] Webserver started at http://127.3.204.126:33813/ using document root <none> and password file <none>
I20260811 12:33:53.050834 13069 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:53.050943 13069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:53.051261 13069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:53.054230 13069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "bf80792fbfa542dbbf0fe93f7097de45"
format_stamp: "Formatted at 2026-08-11 12:33:53 on dist-test-slave-lgr5"
server_key: "ec8c87e82cc4971f4f9c087a5e67e75c"
server_key_iv: "7a17bbb5a843d2ba8533b8804dd130aa"
server_key_version: "encryptionkey@0"
I20260811 12:33:53.055070 13069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "bf80792fbfa542dbbf0fe93f7097de45"
format_stamp: "Formatted at 2026-08-11 12:33:53 on dist-test-slave-lgr5"
server_key: "ec8c87e82cc4971f4f9c087a5e67e75c"
server_key_iv: "7a17bbb5a843d2ba8533b8804dd130aa"
server_key_version: "encryptionkey@0"
I20260811 12:33:53.060510 13069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:33:53.064036 13094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:53.065539 13069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:33:53.065716 13069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "bf80792fbfa542dbbf0fe93f7097de45"
format_stamp: "Formatted at 2026-08-11 12:33:53 on dist-test-slave-lgr5"
server_key: "ec8c87e82cc4971f4f9c087a5e67e75c"
server_key_iv: "7a17bbb5a843d2ba8533b8804dd130aa"
server_key_version: "encryptionkey@0"
I20260811 12:33:53.065935 13069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:53.079650 13069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:53.080605 13069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:53.080915 13069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:53.098330 13069 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:43803
I20260811 12:33:53.098349 13181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:43803 every 8 connection(s)
I20260811 12:33:53.099754 13069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:53.103653 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:53.106508 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 13069
I20260811 12:33:53.106703 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:53.107528 3889 external_mini_cluster.cc:1428] Setting key c6a6adc206eebd3565b62250744dcd76
I20260811 12:33:53.116122 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45: Bootstrap starting.
I20260811 12:33:53.119634 13182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:53.120867 13182 log.cc:824] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:53.121723 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:53.109733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.3.204.126:43803)
0811 12:33:53.110848 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:53.110861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:53.110905 (+ 44us) server_negotiation.cc:407] Connection header received
0811 12:33:53.112662 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:53.112690 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:53.113200 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:53.113616 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:53.114199 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.115502 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.116906 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.117659 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.117839 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:53.119040 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:53.119048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:53.119064 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:53.119660 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:53.119778 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:53.119818 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:53.119930 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:53.120197 (+ 267us) server_negotiation.cc:299] Negotiation successful
0811 12:33:53.120532 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":387,"threads_started":1}
I20260811 12:33:53.124322 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45: No bootstrap required, opened a new log
I20260811 12:33:53.125958 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:33:53.132360 13182 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf80792fbfa542dbbf0fe93f7097de45" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43803 } }
I20260811 12:33:53.132879 13182 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:53.132984 13182 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf80792fbfa542dbbf0fe93f7097de45, State: Initialized, Role: FOLLOWER
I20260811 12:33:53.133579 13182 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf80792fbfa542dbbf0fe93f7097de45" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43803 } }
I20260811 12:33:53.133822 13182 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:53.133977 13182 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:53.134128 13182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:53.136480 13182 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf80792fbfa542dbbf0fe93f7097de45" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43803 } }
I20260811 12:33:53.137059 13182 leader_election.cc:302] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf80792fbfa542dbbf0fe93f7097de45; no voters:
I20260811 12:33:53.137645 13182 leader_election.cc:288] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:53.137806 13188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:53.139276 13188 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [term 1 LEADER]: Becoming Leader. State: Replica: bf80792fbfa542dbbf0fe93f7097de45, State: Running, Role: LEADER
I20260811 12:33:53.139864 13188 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf80792fbfa542dbbf0fe93f7097de45" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43803 } }
I20260811 12:33:53.140184 13182 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:53.143029 13192 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf80792fbfa542dbbf0fe93f7097de45. Latest consensus state: current_term: 1 leader_uuid: "bf80792fbfa542dbbf0fe93f7097de45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf80792fbfa542dbbf0fe93f7097de45" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43803 } } }
I20260811 12:33:53.143055 13190 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf80792fbfa542dbbf0fe93f7097de45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf80792fbfa542dbbf0fe93f7097de45" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43803 } } }
I20260811 12:33:53.143599 13192 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:53.143641 13190 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:53.145309 13203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:53.150458 13203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:53.159720 13203 catalog_manager.cc:1347] Generated new cluster ID: bb0b7994bdd742d089eb0a07b1023d9f
I20260811 12:33:53.159842 13203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:53.171696 13203 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:53.173730 13203 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:53.185565 13203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf80792fbfa542dbbf0fe93f7097de45: Generated new TSK 0
I20260811 12:33:53.186498 13203 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 12:33:53.232261 13231 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260811 12:33:53.232385 13231 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 12:33:53.233839 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:53.210408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.3.204.126:43803)
0811 12:33:53.210634 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:53.210642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:53.211236 (+ 594us) server_negotiation.cc:407] Connection header received
0811 12:33:53.211441 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:53.211445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:53.211529 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:53.211670 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0811 12:33:53.212277 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.213032 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.215113 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.215596 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.215708 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:53.216336 (+ 628us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 12:33:53.233078 (+ 16742us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 12:33:53.233520 (+ 442us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260811 12:33:53.233922 13230 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:33:53.210379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43803 (local address 127.0.0.1:33074)
0811 12:33:53.211045 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:53.211070 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:53.211238 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:53.211706 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:53.211711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:53.211724 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0811 12:33:53.212073 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:53.212115 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.213195 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.213201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:53.214917 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:53.214935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.215776 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.215783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:53.216010 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:53.216061 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 12:33:53.233383 (+ 17322us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 12:33:53.233610 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.204.126:43803: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":322,"threads_started":1}
W20260811 12:33:53.233985 13185 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260811 12:33:53.234647 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 13069
I20260811 12:33:53.235880 13173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:53.332960 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 13069
2026-08-11T12:33:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (753 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260811 12:33:53.395565 3889 mini_oidc.cc:152] Starting JWKS server
I20260811 12:33:53.397619 3889 webserver.cc:460] Webserver started at https://127.0.0.1:41225/ using document root <none> and password file <none>
I20260811 12:33:53.397871 3889 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 12:33:53.398602 3889 webserver.cc:460] Webserver started at http://127.0.0.1:36249/ using document root <none> and password file <none>
2026-08-11T12:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:53Z Disabled control of system clock
I20260811 12:33:53.432560 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:44819
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:44819
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41225/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 12:33:53.772986 13242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:53.773324 13242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:53.773453 13242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:53.780947 13242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:53.781097 13242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:53.781167 13242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:53.781243 13242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:53.781304 13242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 12:33:53.781394 13242 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 12:33:53.781445 13242 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41225/jwks.json
I20260811 12:33:53.791065 13242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:44819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41225/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:53.792486 13242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:53.794319 13242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:53.806236 13253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:53.806329 13252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:53.806270 13259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:53.807498 13242 server_base.cc:1034] running on GCE node
I20260811 12:33:53.808338 13242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:53.809788 13242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:53.811013 13242 hybrid_clock.cc:648] HybridClock initialized: now 1786451633810958 us; error 38 us; skew 500 ppm
I20260811 12:33:53.811396 13242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:33:53.813699 13242 webserver.cc:460] Webserver started at http://127.3.204.126:43525/ using document root <none> and password file <none>
I20260811 12:33:53.814416 13242 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:53.814517 13242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:53.814849 13242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:53.817902 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "dc6746c3ee3f4e13833f07ca1878fafa"
format_stamp: "Formatted at 2026-08-11 12:33:53 on dist-test-slave-lgr5"
server_key: "46b45078fc729a7f2fe33730bf2c7e04"
server_key_iv: "da76954c12a8495c1470e85104a20501"
server_key_version: "encryptionkey@0"
I20260811 12:33:53.818723 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "dc6746c3ee3f4e13833f07ca1878fafa"
format_stamp: "Formatted at 2026-08-11 12:33:53 on dist-test-slave-lgr5"
server_key: "46b45078fc729a7f2fe33730bf2c7e04"
server_key_iv: "da76954c12a8495c1470e85104a20501"
server_key_version: "encryptionkey@0"
I20260811 12:33:53.824209 13242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260811 12:33:53.827968 13267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:53.829512 13242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:53.829703 13242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "dc6746c3ee3f4e13833f07ca1878fafa"
format_stamp: "Formatted at 2026-08-11 12:33:53 on dist-test-slave-lgr5"
server_key: "46b45078fc729a7f2fe33730bf2c7e04"
server_key_iv: "da76954c12a8495c1470e85104a20501"
server_key_version: "encryptionkey@0"
I20260811 12:33:53.829933 13242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:53.844887 13242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:53.845911 13242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:53.846217 13242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:53.863837 13242 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:44819
I20260811 12:33:53.863857 13359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:44819 every 8 connection(s)
I20260811 12:33:53.865299 13242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:53.869165 13360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:53.875818 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 13242
I20260811 12:33:53.876020 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:53.876663 3889 external_mini_cluster.cc:1428] Setting key 6c9e7a52d658b05505c91d1a9506542e
I20260811 12:33:53.880308 13360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa: Bootstrap starting.
I20260811 12:33:53.884552 13360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:53.885883 13360 log.cc:824] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:53.889464 13360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa: No bootstrap required, opened a new log
I20260811 12:33:53.893256 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:53.878745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.3.204.126:44819)
0811 12:33:53.879849 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:53.879865 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:53.879921 (+ 56us) server_negotiation.cc:407] Connection header received
0811 12:33:53.882365 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:53.882397 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:53.883042 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:53.883495 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:53.884062 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.886065 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.887899 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.888968 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.889158 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:53.890318 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:53.890329 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:53.890348 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:53.891052 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:53.891155 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:53.891181 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:53.891295 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:53.891552 (+ 257us) server_negotiation.cc:299] Negotiation successful
0811 12:33:53.891906 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":327,"threads_started":1}
I20260811 12:33:53.897068 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:33:53.898278 13360 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6746c3ee3f4e13833f07ca1878fafa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44819 } }
I20260811 12:33:53.898618 13360 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:53.898707 13360 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6746c3ee3f4e13833f07ca1878fafa, State: Initialized, Role: FOLLOWER
I20260811 12:33:53.899302 13360 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6746c3ee3f4e13833f07ca1878fafa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44819 } }
I20260811 12:33:53.899554 13360 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:53.899684 13360 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:53.899850 13360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:53.902149 13360 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6746c3ee3f4e13833f07ca1878fafa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44819 } }
I20260811 12:33:53.902688 13360 leader_election.cc:302] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6746c3ee3f4e13833f07ca1878fafa; no voters:
I20260811 12:33:53.903230 13360 leader_election.cc:288] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:53.903375 13368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:53.904069 13368 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [term 1 LEADER]: Becoming Leader. State: Replica: dc6746c3ee3f4e13833f07ca1878fafa, State: Running, Role: LEADER
I20260811 12:33:53.904582 13368 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6746c3ee3f4e13833f07ca1878fafa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44819 } }
I20260811 12:33:53.905367 13360 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:53.907449 13370 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6746c3ee3f4e13833f07ca1878fafa. Latest consensus state: current_term: 1 leader_uuid: "dc6746c3ee3f4e13833f07ca1878fafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6746c3ee3f4e13833f07ca1878fafa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44819 } } }
I20260811 12:33:53.907478 13369 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc6746c3ee3f4e13833f07ca1878fafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6746c3ee3f4e13833f07ca1878fafa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44819 } } }
I20260811 12:33:53.907934 13370 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:53.907934 13369 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:53.910307 13376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:53.915431 13376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:53.924571 13376 catalog_manager.cc:1347] Generated new cluster ID: 5c415b4df79b40aa83a23d9888b22a88
I20260811 12:33:53.924703 13376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:53.939936 13376 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:53.941776 13376 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:53.953562 13376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6746c3ee3f4e13833f07ca1878fafa: Generated new TSK 0
I20260811 12:33:53.954720 13376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:33:53.985505 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:53.975633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.3.204.126:44819)
0811 12:33:53.975925 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:53.975933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:53.977466 (+ 1533us) server_negotiation.cc:407] Connection header received
0811 12:33:53.977845 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:53.977852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:53.977970 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:53.978104 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0811 12:33:53.978922 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.979663 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.981690 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.982167 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.982274 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:53.982884 (+ 610us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 12:33:53.984839 (+ 1955us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 12:33:53.985166 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":99}
W20260811 12:33:53.985646 13363 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260811 12:33:53.985736 13403 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:33:53.975669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:44819 (local address 127.0.0.1:35222)
0811 12:33:53.977347 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:53.977367 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:53.977596 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:53.978357 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:53.978362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:53.978376 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0811 12:33:53.978717 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:53.978729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.979816 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.979822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:53.981529 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:53.981547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:53.982364 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:53.982371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:53.982561 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:53.982611 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 12:33:53.985044 (+ 2433us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 12:33:53.985303 (+ 259us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.204.126:44819: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":1514,"thread_start_us":1363,"threads_started":1}
I20260811 12:33:53.986584 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 13242
I20260811 12:33:53.987695 13350 generic_service.cc:196] Checking for leaks (request via RPC)
E20260811 12:33:54.001523 13405 webserver.cc:533] Webserver: sslize failed:
E20260811 12:33:54.001668 13405 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 12:33:54.083587 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 13242
2026-08-11T12:33:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (753 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-11T12:33:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:54Z Disabled control of system clock
I20260811 12:33:54.180485 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:44993
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:44993
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/testchainca.cert with env {}
W20260811 12:33:54.516521 13410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:54.516837 13410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:54.516938 13410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:54.524318 13410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:54.524441 13410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:54.524477 13410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:54.524535 13410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:54.524578 13410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 12:33:54.524641 13410 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 12:33:54.524680 13410 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260811 12:33:54.534889 13410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:44993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:44993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:54.536269 13410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:54.537560 13410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:54.549093 13418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:54.549049 13419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:54.549137 13423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:54.549623 13410 server_base.cc:1034] running on GCE node
I20260811 12:33:54.550532 13410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:54.551949 13410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:54.553177 13410 hybrid_clock.cc:648] HybridClock initialized: now 1786451634553125 us; error 43 us; skew 500 ppm
I20260811 12:33:54.553555 13410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:33:54.555950 13410 webserver.cc:460] Webserver started at http://127.3.204.126:41649/ using document root <none> and password file <none>
I20260811 12:33:54.556717 13410 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:54.556819 13410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:54.557158 13410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:54.560221 13410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "4bc9560b09014645b4265ea001c7d925"
format_stamp: "Formatted at 2026-08-11 12:33:54 on dist-test-slave-lgr5"
server_key: "6bb268789d10940c717f62f57bdbb959"
server_key_iv: "adc39a7fc3bc5209febfb7391d7b0dc2"
server_key_version: "encryptionkey@0"
I20260811 12:33:54.561064 13410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "4bc9560b09014645b4265ea001c7d925"
format_stamp: "Formatted at 2026-08-11 12:33:54 on dist-test-slave-lgr5"
server_key: "6bb268789d10940c717f62f57bdbb959"
server_key_iv: "adc39a7fc3bc5209febfb7391d7b0dc2"
server_key_version: "encryptionkey@0"
I20260811 12:33:54.566337 13410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:33:54.569936 13432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:54.571245 13410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:33:54.571408 13410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "4bc9560b09014645b4265ea001c7d925"
format_stamp: "Formatted at 2026-08-11 12:33:54 on dist-test-slave-lgr5"
server_key: "6bb268789d10940c717f62f57bdbb959"
server_key_iv: "adc39a7fc3bc5209febfb7391d7b0dc2"
server_key_version: "encryptionkey@0"
I20260811 12:33:54.571620 13410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:54.586282 13410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:54.587144 13410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:54.587417 13410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:54.605000 13410 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:44993
I20260811 12:33:54.605006 13526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:44993 every 8 connection(s)
I20260811 12:33:54.606346 13410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:54.610078 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:54.612895 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 13410
I20260811 12:33:54.613072 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:54.613727 3889 external_mini_cluster.cc:1428] Setting key 41984252b73abe265b5548df51f19373
I20260811 12:33:54.623560 13527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925: Bootstrap starting.
I20260811 12:33:54.627386 13527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:54.628762 13527 log.cc:824] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:54.629263 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:54.615728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.3.204.126:44993)
0811 12:33:54.616851 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:54.616872 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:54.616931 (+ 59us) server_negotiation.cc:407] Connection header received
0811 12:33:54.618965 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:54.618999 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:54.619678 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:54.620197 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:54.620775 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:54.622699 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:54.624119 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:54.625101 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:54.625319 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:54.626388 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:54.626398 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:54.626422 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:33:54.627111 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:54.627235 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:33:54.627275 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:54.627397 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:54.627603 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 12:33:54.627957 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":342,"threads_started":1}
I20260811 12:33:54.631896 13527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925: No bootstrap required, opened a new log
I20260811 12:33:54.633137 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:33:54.640146 13527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9560b09014645b4265ea001c7d925" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44993 } }
I20260811 12:33:54.640525 13527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:54.640625 13527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc9560b09014645b4265ea001c7d925, State: Initialized, Role: FOLLOWER
I20260811 12:33:54.641278 13527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9560b09014645b4265ea001c7d925" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44993 } }
I20260811 12:33:54.641489 13527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:54.641585 13527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:54.641705 13527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:54.643827 13527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9560b09014645b4265ea001c7d925" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44993 } }
I20260811 12:33:54.644445 13527 leader_election.cc:302] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bc9560b09014645b4265ea001c7d925; no voters:
I20260811 12:33:54.645108 13527 leader_election.cc:288] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:54.645237 13535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:54.646229 13535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc9560b09014645b4265ea001c7d925, State: Running, Role: LEADER
I20260811 12:33:54.646749 13535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9560b09014645b4265ea001c7d925" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44993 } }
I20260811 12:33:54.647285 13527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:54.649679 13537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc9560b09014645b4265ea001c7d925. Latest consensus state: current_term: 1 leader_uuid: "4bc9560b09014645b4265ea001c7d925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9560b09014645b4265ea001c7d925" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44993 } } }
I20260811 12:33:54.649690 13536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bc9560b09014645b4265ea001c7d925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9560b09014645b4265ea001c7d925" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44993 } } }
I20260811 12:33:54.650087 13536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:54.650087 13537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:54.651880 13547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:54.657598 13547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:54.666251 13547 catalog_manager.cc:1347] Generated new cluster ID: 3c488c55c3a148378d4ce093eb163b6c
I20260811 12:33:54.666375 13547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:54.682371 13547 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:54.683925 13547 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:54.691623 13547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bc9560b09014645b4265ea001c7d925: Generated new TSK 0
I20260811 12:33:54.692530 13547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:33:55.164896 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:54.704025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.3.204.126:44993)
0811 12:33:54.704245 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:54.704251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:54.704609 (+ 358us) server_negotiation.cc:407] Connection header received
0811 12:33:54.704821 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:54.704827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:54.704925 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:54.705126 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0811 12:33:54.705534 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:54.706668 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:54.708206 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:54.708904 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:54.709054 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:54.709563 (+ 509us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 12:33:55.164290 (+454727us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 12:33:55.164666 (+ 376us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53656: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260811 12:33:55.165009 13531 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53656: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260811 12:33:55.165226 13576 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:33:54.703855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:44993 (local address 127.0.0.1:53656)
0811 12:33:54.704512 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:54.704528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:54.704650 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:54.705113 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:54.705116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:54.705125 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0811 12:33:54.705384 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:54.705391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:54.706876 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:54.706881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:54.708006 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:54.708020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:54.709081 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:54.709085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:54.709214 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:54.709257 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 12:33:55.164606 (+455349us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 12:33:55.164868 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.204.126:44993: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":374,"threads_started":1}
I20260811 12:33:55.166395 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 13410
I20260811 12:33:55.167574 13515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:55.270699 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 13410
2026-08-11T12:33:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1165 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (423 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (428 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:33:56 dist-test-slave-lgr5 krb5kdc[13597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:33:56 dist-test-slave-lgr5 krb5kdc[13597](info): set up 2 sockets
Aug 11 12:33:56 dist-test-slave-lgr5 krb5kdc[13597](info): commencing operation
krb5kdc: starting...
W20260811 12:33:57.372601 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:33:57 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:33:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:33:57.609177 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37783
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:37783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:57.945631 13616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:57.945962 13616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:57.946106 13616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:57.953420 13616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:33:57.953548 13616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:57.953629 13616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:57.953706 13616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:33:57.953766 13616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:33:57.963932 13616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:37783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:57.965382 13616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:57.966782 13616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:57.980813 13629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:57.980871 13625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:57.981604 13616 server_base.cc:1034] running on GCE node
W20260811 12:33:57.982012 13626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:57.982944 13616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:57.984460 13616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:57.985704 13616 hybrid_clock.cc:648] HybridClock initialized: now 1786451637985634 us; error 49 us; skew 500 ppm
I20260811 12:33:57.986126 13616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:57 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:57.992865 13616 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:33:57.994539 13616 webserver.cc:460] Webserver started at http://127.3.204.126:39577/ using document root <none> and password file <none>
I20260811 12:33:57.995301 13616 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:57.995450 13616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:57.995781 13616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:57.999101 13616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "95b8b54356de4b29a14bc20f6de87906"
format_stamp: "Formatted at 2026-08-11 12:33:57 on dist-test-slave-lgr5"
server_key: "3ebd83f9fc9173ff8c845ccbc3ae5a4f"
server_key_iv: "8ce1cc6721bce4936b4cd4c948eb4dff"
server_key_version: "encryptionkey@0"
I20260811 12:33:57.999890 13616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "95b8b54356de4b29a14bc20f6de87906"
format_stamp: "Formatted at 2026-08-11 12:33:57 on dist-test-slave-lgr5"
server_key: "3ebd83f9fc9173ff8c845ccbc3ae5a4f"
server_key_iv: "8ce1cc6721bce4936b4cd4c948eb4dff"
server_key_version: "encryptionkey@0"
I20260811 12:33:58.005438 13616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260811 12:33:58.009106 13640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:58.010546 13616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:33:58.010725 13616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "95b8b54356de4b29a14bc20f6de87906"
format_stamp: "Formatted at 2026-08-11 12:33:57 on dist-test-slave-lgr5"
server_key: "3ebd83f9fc9173ff8c845ccbc3ae5a4f"
server_key_iv: "8ce1cc6721bce4936b4cd4c948eb4dff"
server_key_version: "encryptionkey@0"
I20260811 12:33:58.010919 13616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:58.040970 13616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:58.045207 13616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:58.045487 13616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:58.063696 13616 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:37783
I20260811 12:33:58.063719 13731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:37783 every 8 connection(s)
I20260811 12:33:58.065204 13616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:33:58.069233 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:33:58.073556 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 13616
I20260811 12:33:58.073787 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:33:58.074591 3889 external_mini_cluster.cc:1428] Setting key 1497a9d3d6bb59d5a6ae76e1e9847065
I20260811 12:33:58.081525 13732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906: Bootstrap starting.
I20260811 12:33:58.085278 13732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906: Neither blocks nor log segments found. Creating new log.
I20260811 12:33:58.086510 13732 log.cc:824] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906: Log is configured to *not* fsync() on all Append() calls
I20260811 12:33:58.090065 13732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906: No bootstrap required, opened a new log
Aug 11 12:33:58 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:58.098112 13732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b8b54356de4b29a14bc20f6de87906" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37783 } }
I20260811 12:33:58.098517 13732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:33:58.098675 13732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b8b54356de4b29a14bc20f6de87906, State: Initialized, Role: FOLLOWER
I20260811 12:33:58.099144 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:58.076935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.3.204.126:37783)
0811 12:33:58.077740 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:58.077752 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:58.077799 (+ 47us) server_negotiation.cc:407] Connection header received
0811 12:33:58.079914 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:58.079944 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:58.080428 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:58.080797 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:58.082594 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:58.083940 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:58.085688 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:58.086435 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:58.086591 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:58.091182 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:58.091191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:58.091206 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:58.091250 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:58.094521 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:58.095301 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:58.095310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:58.095317 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:58.095527 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:58.095949 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:58.095953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:58.095955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:58.096646 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:58.096991 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:58.097452 (+ 461us) server_negotiation.cc:299] Negotiation successful
0811 12:33:58.097894 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":371,"threads_started":1}
I20260811 12:33:58.099334 13732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b8b54356de4b29a14bc20f6de87906" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37783 } }
I20260811 12:33:58.099593 13732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:33:58.099741 13732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:33:58.099920 13732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:33:58.102399 13732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b8b54356de4b29a14bc20f6de87906" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37783 } }
I20260811 12:33:58.102876 13732 leader_election.cc:302] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b8b54356de4b29a14bc20f6de87906; no voters:
I20260811 12:33:58.103489 13732 leader_election.cc:288] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:33:58.103586 13739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:33:58.104887 13739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [term 1 LEADER]: Becoming Leader. State: Replica: 95b8b54356de4b29a14bc20f6de87906, State: Running, Role: LEADER
I20260811 12:33:58.105406 13739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b8b54356de4b29a14bc20f6de87906" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37783 } }
I20260811 12:33:58.105705 13732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:33:58.108449 13740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95b8b54356de4b29a14bc20f6de87906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b8b54356de4b29a14bc20f6de87906" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37783 } } }
I20260811 12:33:58.108422 13741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b8b54356de4b29a14bc20f6de87906. Latest consensus state: current_term: 1 leader_uuid: "95b8b54356de4b29a14bc20f6de87906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b8b54356de4b29a14bc20f6de87906" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37783 } } }
I20260811 12:33:58.108906 13741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:58.108911 13740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906 [sys.catalog]: This master's current role is: LEADER
I20260811 12:33:58.110749 13751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:33:58.116744 13751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:33:58.125819 13751 catalog_manager.cc:1347] Generated new cluster ID: 92d8e57dcec844f59ad27637f14fb1ac
I20260811 12:33:58.125967 13751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:33:58.137840 13751 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:33:58.139427 13751 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:33:58.146544 13751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95b8b54356de4b29a14bc20f6de87906: Generated new TSK 0
I20260811 12:33:58.147426 13751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:33:58.212544 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37783
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:58.555080 13775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:58.555454 13775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:58.555606 13775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:58.563716 13775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:58.563864 13775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:58.564023 13775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:33:58.574275 13775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:58.575693 13775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:58.577092 13775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:58.590859 13782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:58.590859 13783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:58.590867 13786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:58.591629 13775 server_base.cc:1034] running on GCE node
I20260811 12:33:58.592417 13775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:58.593466 13775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:58.594705 13775 hybrid_clock.cc:648] HybridClock initialized: now 1786451638594649 us; error 47 us; skew 500 ppm
I20260811 12:33:58.595080 13775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:58 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:58.598619 13775 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:33:58.600334 13775 webserver.cc:460] Webserver started at http://127.3.204.65:40867/ using document root <none> and password file <none>
I20260811 12:33:58.601065 13775 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:58.601181 13775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:58.601511 13775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:58.604727 13775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "09b8b8efa58d48abb02414c28dcdef24"
format_stamp: "Formatted at 2026-08-11 12:33:58 on dist-test-slave-lgr5"
server_key: "7b4f832bf7224064c6aba084bc1b70f7"
server_key_iv: "97fb1ec975b8f99df0bb09a578b0f45a"
server_key_version: "encryptionkey@0"
I20260811 12:33:58.605541 13775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "09b8b8efa58d48abb02414c28dcdef24"
format_stamp: "Formatted at 2026-08-11 12:33:58 on dist-test-slave-lgr5"
server_key: "7b4f832bf7224064c6aba084bc1b70f7"
server_key_iv: "97fb1ec975b8f99df0bb09a578b0f45a"
server_key_version: "encryptionkey@0"
I20260811 12:33:58.610947 13775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:58.614569 13795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:58.615983 13775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 12:33:58.616160 13775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "09b8b8efa58d48abb02414c28dcdef24"
format_stamp: "Formatted at 2026-08-11 12:33:58 on dist-test-slave-lgr5"
server_key: "7b4f832bf7224064c6aba084bc1b70f7"
server_key_iv: "97fb1ec975b8f99df0bb09a578b0f45a"
server_key_version: "encryptionkey@0"
I20260811 12:33:58.616360 13775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:58.633118 13775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:58.637475 13775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:58.637750 13775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:58.638738 13775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:58.640127 13775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:58.640204 13775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:58.640328 13775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:58.640396 13775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:58.671479 13775 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:34427
I20260811 12:33:58.671535 13982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:34427 every 8 connection(s)
I20260811 12:33:58.672847 13775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:33:58.678997 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 13775
I20260811 12:33:58.679205 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:33:58.680066 3889 external_mini_cluster.cc:1428] Setting key 5165a901dd086a4eec818aae96315add
Aug 11 12:33:58 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:58.695205 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:58.675462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:37973 (local address 127.3.204.126:37783)
0811 12:33:58.675949 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:58.675957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:58.676985 (+ 1028us) server_negotiation.cc:407] Connection header received
0811 12:33:58.678943 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:58.678948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:58.679049 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:58.679189 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:58.681278 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:58.682135 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:58.683925 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:58.684608 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:58.684763 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:58.688531 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:58.688538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:58.688543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:58.688591 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:58.691663 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:58.692704 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:58.692709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:58.692711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:58.692847 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:58.693316 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:58.693319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:58.693321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:58.693624 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:58.693743 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:58.694724 (+ 981us) server_negotiation.cc:299] Negotiation successful
0811 12:33:58.694913 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260811 12:33:58.696699 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:58.676014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37783 (local address 127.3.204.65:37973)
0811 12:33:58.676797 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:58.676851 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:58.678564 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:58.679520 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:58.679534 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:58.680371 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:58.681107 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:58.681126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:58.682329 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:58.682335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:58.683744 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:58.683758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:58.684844 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:58.684849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:58.685082 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:58.685767 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:58.685796 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:58.688266 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:58.691858 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:58.691870 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:58.691892 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:58.692537 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:58.692957 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:58.692962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:58.692965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:58.693217 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:58.693756 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:58.693764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:58.694301 (+ 537us) client_negotiation.cc:769] Sending connection context
0811 12:33:58.694616 (+ 315us) client_negotiation.cc:240] Negotiation successful
0811 12:33:58.695194 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":310,"threads_started":1}
I20260811 12:33:58.698347 13983 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37783
I20260811 12:33:58.698674 13983 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:58.699647 13983 heartbeater.cc:511] Master 127.3.204.126:37783 requested a full tablet report, sending...
I20260811 12:33:58.701915 13671 ts_manager.cc:194] Registered new tserver with Master: 09b8b8efa58d48abb02414c28dcdef24 (127.3.204.65:34427)
I20260811 12:33:58.703441 13671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:37973
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:33:58.735910 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37783
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:59.076884 13991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:59.077190 13991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:59.077327 13991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:59.084724 13991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:59.084863 13991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:59.085026 13991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:33:59.095093 13991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:59.096520 13991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:59.097873 13991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:59.112164 13997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:59.112164 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:59.112191 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:59.112916 13991 server_base.cc:1034] running on GCE node
I20260811 12:33:59.113628 13991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:59.114765 13991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:59.116014 13991 hybrid_clock.cc:648] HybridClock initialized: now 1786451639115938 us; error 58 us; skew 500 ppm
I20260811 12:33:59.116385 13991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.119853 13991 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:33:59.121579 13991 webserver.cc:460] Webserver started at http://127.3.204.66:37991/ using document root <none> and password file <none>
I20260811 12:33:59.122314 13991 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:59.122437 13991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:59.122759 13991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:59.125885 13991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "85f1cf88b99945b0a8fd79e365f74e1b"
format_stamp: "Formatted at 2026-08-11 12:33:59 on dist-test-slave-lgr5"
server_key: "8cf65e12f884e80a75a2d92a8a44ddb8"
server_key_iv: "6821f72ce66492cd9276000d36e5cbb9"
server_key_version: "encryptionkey@0"
I20260811 12:33:59.126675 13991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "85f1cf88b99945b0a8fd79e365f74e1b"
format_stamp: "Formatted at 2026-08-11 12:33:59 on dist-test-slave-lgr5"
server_key: "8cf65e12f884e80a75a2d92a8a44ddb8"
server_key_iv: "6821f72ce66492cd9276000d36e5cbb9"
server_key_version: "encryptionkey@0"
I20260811 12:33:59.132417 13991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260811 12:33:59.135919 14015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:59.137293 13991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:59.137457 13991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "85f1cf88b99945b0a8fd79e365f74e1b"
format_stamp: "Formatted at 2026-08-11 12:33:59 on dist-test-slave-lgr5"
server_key: "8cf65e12f884e80a75a2d92a8a44ddb8"
server_key_iv: "6821f72ce66492cd9276000d36e5cbb9"
server_key_version: "encryptionkey@0"
I20260811 12:33:59.137656 13991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:59.158638 13991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:59.163311 13991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:59.163609 13991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:59.164713 13991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:59.166145 13991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:59.166234 13991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:59.166364 13991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:59.166422 13991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:59.197772 13991 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:35979
I20260811 12:33:59.197842 14208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:35979 every 8 connection(s)
I20260811 12:33:59.199124 13991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:33:59.200387 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 13991
I20260811 12:33:59.200580 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:33:59.201191 3889 external_mini_cluster.cc:1428] Setting key a6dc7438d2aec2205f88f300a06ef792
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:59.221102 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.202064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:38725 (local address 127.3.204.126:37783)
0811 12:33:59.202576 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.202582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.203652 (+ 1070us) server_negotiation.cc:407] Connection header received
0811 12:33:59.205546 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.205552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.205635 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.205729 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.207322 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.208059 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.210137 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.210592 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.210688 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.214542 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:59.214550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:59.214555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:59.214598 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:59.217775 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.218537 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.218542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.218545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.218712 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.219111 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.219115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.219118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.219520 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:59.219682 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:59.220451 (+ 769us) server_negotiation.cc:299] Negotiation successful
0811 12:33:59.220710 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":243,"threads_started":1}
I20260811 12:33:59.222010 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.202673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37783 (local address 127.3.204.66:38725)
0811 12:33:59.203500 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:59.203546 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:59.205253 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:59.205996 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:59.206005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:59.206573 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:59.207150 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:59.207169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.208226 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.208231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:59.209987 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:59.210003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.210817 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.210823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:59.211103 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.211778 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:59.211807 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:59.214244 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:59.217932 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:59.217939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:59.217955 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:59.218406 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:59.218806 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:59.218809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:59.218811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:59.219005 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:59.219643 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:59.219649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:59.220005 (+ 356us) client_negotiation.cc:769] Sending connection context
0811 12:33:59.220338 (+ 333us) client_negotiation.cc:240] Negotiation successful
0811 12:33:59.220772 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":342,"threads_started":1}
I20260811 12:33:59.223476 14210 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37783
I20260811 12:33:59.223831 14210 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:59.224825 14210 heartbeater.cc:511] Master 127.3.204.126:37783 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
I20260811 12:33:59.226643 13671 ts_manager.cc:194] Registered new tserver with Master: 85f1cf88b99945b0a8fd79e365f74e1b (127.3.204.66:35979)
I20260811 12:33:59.227558 13671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:38725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:33:59.257530 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37783
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:33:59.592283 14216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:33:59.592587 14216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:33:59.592708 14216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:33:59.600209 14216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:33:59.600327 14216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:33:59.600479 14216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:33:59.610968 14216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:33:59.612388 14216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:33:59.613745 14216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:33:59.627656 14223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:59.627588 14224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:33:59.627611 14227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:33:59.628665 14216 server_base.cc:1034] running on GCE node
I20260811 12:33:59.629354 14216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:33:59.630342 14216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:33:59.631547 14216 hybrid_clock.cc:648] HybridClock initialized: now 1786451639631486 us; error 50 us; skew 500 ppm
I20260811 12:33:59.631861 14216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.635095 14216 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:33:59.636577 14216 webserver.cc:460] Webserver started at http://127.3.204.67:34913/ using document root <none> and password file <none>
I20260811 12:33:59.637282 14216 fs_manager.cc:359] Metadata directory not provided
I20260811 12:33:59.637396 14216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:33:59.637714 14216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:33:59.640802 14216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "2ed933d60cc04072bd2cb81ad4e0c152"
format_stamp: "Formatted at 2026-08-11 12:33:59 on dist-test-slave-lgr5"
server_key: "36e3a892e8331615a9d218e2dd824495"
server_key_iv: "bff3bd05f1fabc9652d05cc0af423174"
server_key_version: "encryptionkey@0"
I20260811 12:33:59.641710 14216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "2ed933d60cc04072bd2cb81ad4e0c152"
format_stamp: "Formatted at 2026-08-11 12:33:59 on dist-test-slave-lgr5"
server_key: "36e3a892e8331615a9d218e2dd824495"
server_key_iv: "bff3bd05f1fabc9652d05cc0af423174"
server_key_version: "encryptionkey@0"
I20260811 12:33:59.647410 14216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:33:59.650954 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:59.652386 14216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:33:59.652562 14216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "2ed933d60cc04072bd2cb81ad4e0c152"
format_stamp: "Formatted at 2026-08-11 12:33:59 on dist-test-slave-lgr5"
server_key: "36e3a892e8331615a9d218e2dd824495"
server_key_iv: "bff3bd05f1fabc9652d05cc0af423174"
server_key_version: "encryptionkey@0"
I20260811 12:33:59.652760 14216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:33:59.668565 14216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:33:59.673715 14216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:33:59.674049 14216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:33:59.675287 14216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:33:59.677047 14216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:33:59.677143 14216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:59.677268 14216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:33:59.677331 14216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:33:59.707063 13983 heartbeater.cc:503] Master 127.3.204.126:37783 was elected leader, sending a full tablet report...
I20260811 12:33:59.708729 14216 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:33761
I20260811 12:33:59.708732 14432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:33761 every 8 connection(s)
I20260811 12:33:59.710114 14216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:33:59.710915 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 14216
I20260811 12:33:59.711086 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:33:59.711653 3889 external_mini_cluster.cc:1428] Setting key 1cc982b8c2193c3f83f832c8f7a86ebf
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:33:59.730454 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.713050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:59383 (local address 127.3.204.126:37783)
0811 12:33:59.713218 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.713223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.714575 (+ 1352us) server_negotiation.cc:407] Connection header received
0811 12:33:59.716479 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.716484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.716565 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.716647 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.718107 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.718847 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.720621 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.721045 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.721151 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.724647 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:59.724652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:59.724655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:59.724690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:59.727389 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.728176 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.728180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.728183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.728315 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.728700 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.728703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.728706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.729010 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:59.729135 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:59.730016 (+ 881us) server_negotiation.cc:299] Negotiation successful
0811 12:33:59.730208 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 12:33:59.731454 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.713558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37783 (local address 127.3.204.67:59383)
0811 12:33:59.714424 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0811 12:33:59.714480 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 12:33:59.716224 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:33:59.716841 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:33:59.716849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:33:59.717376 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:33:59.717959 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:59.717978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.719025 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.719030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:59.720472 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:33:59.720486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.721282 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.721288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:33:59.721507 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.722119 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:33:59.722141 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:33:59.724417 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:33:59.727533 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:59.727540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:59.727555 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:59.728009 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:59.728410 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:33:59.728414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:33:59.728416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:33:59.728613 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:33:59.729163 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:33:59.729171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:33:59.729583 (+ 412us) client_negotiation.cc:769] Sending connection context
0811 12:33:59.729854 (+ 271us) client_negotiation.cc:240] Negotiation successful
0811 12:33:59.730281 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260811 12:33:59.733079 14433 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37783
I20260811 12:33:59.733436 14433 heartbeater.cc:464] Registering TS with master...
I20260811 12:33:59.734251 14433 heartbeater.cc:511] Master 127.3.204.126:37783 requested a full tablet report, sending...
I20260811 12:33:59.735811 13671 ts_manager.cc:194] Registered new tserver with Master: 2ed933d60cc04072bd2cb81ad4e0c152 (127.3.204.67:33761)
I20260811 12:33:59.736686 13671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:59383
I20260811 12:33:59.739331 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/test-admin.keytab.
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.770776 3889 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260811 12:33:59.776182 14443 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:33:59.773617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.3.204.65:34427)
0811 12:33:59.774269 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.774276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.774302 (+ 26us) server_negotiation.cc:407] Connection header received
0811 12:33:59.774365 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.774372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.774581 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.774728 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.775719 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39338: BlockingRecv error: recv got EOF from 127.0.0.1:39338 (error 108)
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":329,"threads_started":1}
W20260811 12:33:59.776475 3889 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5cc' of type FILE
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.778206 3889 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.779624 3889 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:59.799537 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.781369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.3.204.65:34427)
0811 12:33:59.781571 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.781577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.781783 (+ 206us) server_negotiation.cc:407] Connection header received
0811 12:33:59.781940 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.781944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.782021 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.782105 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.783168 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.784214 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.785363 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.785933 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.786035 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.793913 (+ 7878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:59.793926 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:59.793944 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:59.793981 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:59.797005 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.797677 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.797685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.797687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.797809 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.798105 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.798108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.798111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.798475 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:59.798703 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:59.799057 (+ 354us) server_negotiation.cc:299] Negotiation successful
0811 12:33:59.799267 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260811 12:33:59.805792 14443 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:33:59.803862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.3.204.65:34427)
0811 12:33:59.804070 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.804098 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.804281 (+ 183us) server_negotiation.cc:407] Connection header received
0811 12:33:59.804440 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.804444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.804538 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.804625 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.805562 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39354: BlockingRecv error: recv got EOF from 127.0.0.1:39354 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260811 12:33:59.806057 3889 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5cc' of type FILE
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.807685 3889 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.809005 3889 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:59.823709 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.810314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.3.204.65:34427)
0811 12:33:59.810510 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.810516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.810735 (+ 219us) server_negotiation.cc:407] Connection header received
0811 12:33:59.810853 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.810857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.810917 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.810998 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.811933 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.812693 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.813842 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.814260 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.814355 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.818130 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:59.818136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:59.818140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:59.818184 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:59.820866 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.821794 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.821797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.821800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.821930 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.822301 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.822304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.822306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.822595 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:59.822805 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:59.823201 (+ 396us) server_negotiation.cc:299] Negotiation successful
0811 12:33:59.823380 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 12:33:59.828270 14443 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:33:59.826684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.3.204.65:34427)
0811 12:33:59.826923 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.826927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.827003 (+ 76us) server_negotiation.cc:407] Connection header received
0811 12:33:59.827134 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.827137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.827213 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.827294 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.828016 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39380: BlockingRecv error: recv got EOF from 127.0.0.1:39380 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260811 12:33:59.828470 3889 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786451585252647-3889-0/krb5kdc/krb5cc' of type FILE
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.830004 3889 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:33:59.831250 3889 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:33:59.845575 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.832660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.3.204.65:34427)
0811 12:33:59.832886 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.832891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.833077 (+ 186us) server_negotiation.cc:407] Connection header received
0811 12:33:59.833202 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.833206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.833266 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.833343 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.834253 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.835011 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.836442 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.836893 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.837001 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.840607 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:59.840612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:59.840615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:59.840653 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:59.843214 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.843833 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.843836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.843839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.844005 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.844332 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.844335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.844337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.844628 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:59.844857 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:59.845070 (+ 213us) server_negotiation.cc:299] Negotiation successful
0811 12:33:59.845260 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260811 12:33:59.847131 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 13775
I20260811 12:33:59.858911 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.847860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.3.204.65:34427)
0811 12:33:59.847994 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.847998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.848414 (+ 416us) server_negotiation.cc:407] Connection header received
0811 12:33:59.848523 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.848526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.848581 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.848653 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.849608 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.850304 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.851480 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.851924 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.852027 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.853976 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:59.853981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:59.853984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:59.854016 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:59.856595 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.857334 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.857338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.857340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.857502 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.857795 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.857798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.857800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.858079 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:59.858280 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:59.858490 (+ 210us) server_negotiation.cc:299] Negotiation successful
0811 12:33:59.858654 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 12:33:59.859493 13966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:33:59.947970 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 13775
I20260811 12:33:59.976249 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 13991
Aug 11 12:33:59 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:33:59.995813 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:33:59.977281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.3.204.66:35979)
0811 12:33:59.978303 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0811 12:33:59.978313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:33:59.978339 (+ 26us) server_negotiation.cc:407] Connection header received
0811 12:33:59.978404 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:33:59.978411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:33:59.978637 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:33:59.978802 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:33:59.980243 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.981919 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.983679 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:33:59.984607 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:33:59.984743 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:33:59.989804 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:33:59.989825 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:33:59.989848 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:33:59.989915 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:33:59.992898 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.993653 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.993662 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.993665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.993830 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:33:59.994277 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:33:59.994280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:33:59.994282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:33:59.994668 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:33:59.994909 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0811 12:33:59.995255 (+ 346us) server_negotiation.cc:299] Negotiation successful
0811 12:33:59.995515 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":716,"threads_started":1}
I20260811 12:33:59.997263 14204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:00.083926 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 13991
I20260811 12:34:00.106195 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 14216
Aug 11 12:34:00 dist-test-slave-lgr5 krb5kdc[13597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:34:00.122713 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:00.107317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.3.204.67:33761)
0811 12:34:00.108038 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:00.108046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:00.108095 (+ 49us) server_negotiation.cc:407] Connection header received
0811 12:34:00.108172 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:00.108179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:00.108408 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:00.108548 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:00.109598 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:00.110667 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:00.111847 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:00.112733 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:00.112839 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:00.116950 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:00.116964 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:00.116982 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:00.117022 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:00.120034 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:00.120790 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:00.120798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:00.120801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:00.120939 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:00.121279 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:00.121283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:00.121285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:00.121671 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:00.121932 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:00.122126 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 12:34:00.122378 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":390,"threads_started":1}
I20260811 12:34:00.124145 14427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:00.207134 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 14216
I20260811 12:34:00.227587 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 13616
I20260811 12:34:00.228982 13726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:00.332265 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 13616
2026-08-11T12:34:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4218 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:34:00 dist-test-slave-lgr5 krb5kdc[14485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:34:00 dist-test-slave-lgr5 krb5kdc[14485](info): set up 2 sockets
Aug 11 12:34:00 dist-test-slave-lgr5 krb5kdc[14485](info): commencing operation
krb5kdc: starting...
W20260811 12:34:01.595726 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:34:01 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:34:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.126.keytab.
Entry for principal oryx/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.126.keytab.
I20260811 12:34:01.843641 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:36811
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.126.keytab
--principal=oryx/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:36811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:02.179286 14503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:02.179629 14503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:02.179781 14503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:02.187374 14503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:02.187513 14503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:02.187570 14503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:02.187656 14503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:02.187719 14503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:02.198010 14503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:36811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.126.keytab
--principal=oryx/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:36811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:02.199443 14503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:02.200838 14503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:02.212736 14515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:02.212741 14521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:02.212682 14517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:02.213624 14503 server_base.cc:1034] running on GCE node
I20260811 12:34:02.214437 14503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:02.215947 14503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:02.217181 14503 hybrid_clock.cc:648] HybridClock initialized: now 1786451642217117 us; error 51 us; skew 500 ppm
I20260811 12:34:02.217590 14503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:02 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451642, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:02.220832 14503 init.cc:377] Logged in from keytab as oryx/127.3.204.126@KRBTEST.COM (short username oryx)
I20260811 12:34:02.222401 14503 webserver.cc:460] Webserver started at http://127.3.204.126:34285/ using document root <none> and password file <none>
I20260811 12:34:02.223114 14503 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:02.223237 14503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:02.223539 14503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:02.226764 14503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "917d930be6064324b7e8f465196a501e"
format_stamp: "Formatted at 2026-08-11 12:34:02 on dist-test-slave-lgr5"
server_key: "0d51710e4fedc9dd29fec61bc00a4ef0"
server_key_iv: "093ea15be0b407e2579bf52c7ab5ef50"
server_key_version: "encryptionkey@0"
I20260811 12:34:02.227569 14503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "917d930be6064324b7e8f465196a501e"
format_stamp: "Formatted at 2026-08-11 12:34:02 on dist-test-slave-lgr5"
server_key: "0d51710e4fedc9dd29fec61bc00a4ef0"
server_key_iv: "093ea15be0b407e2579bf52c7ab5ef50"
server_key_version: "encryptionkey@0"
I20260811 12:34:02.233119 14503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 12:34:02.236945 14536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:02.238389 14503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:34:02.238570 14503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "917d930be6064324b7e8f465196a501e"
format_stamp: "Formatted at 2026-08-11 12:34:02 on dist-test-slave-lgr5"
server_key: "0d51710e4fedc9dd29fec61bc00a4ef0"
server_key_iv: "093ea15be0b407e2579bf52c7ab5ef50"
server_key_version: "encryptionkey@0"
I20260811 12:34:02.238777 14503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:02.266857 14503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:02.271327 14503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:02.271631 14503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:02.290205 14503 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:36811
I20260811 12:34:02.290223 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:36811 every 8 connection(s)
I20260811 12:34:02.291683 14503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:34:02.295501 14615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:02.297874 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 14503
I20260811 12:34:02.298106 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:34:02.298877 3889 external_mini_cluster.cc:1428] Setting key 277b5b2465c7e3f703d4ec31ea2064da
I20260811 12:34:02.308826 14615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: Bootstrap starting.
I20260811 12:34:02.312896 14615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: Neither blocks nor log segments found. Creating new log.
Aug 11 12:34:02 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:02.314173 14615 log.cc:824] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:02.318005 14615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: No bootstrap required, opened a new log
I20260811 12:34:02.324242 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:02.301177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.3.204.126:36811)
0811 12:34:02.301982 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:02.301998 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:02.302052 (+ 54us) server_negotiation.cc:407] Connection header received
0811 12:34:02.303935 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:02.303960 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:02.304414 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:02.304741 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:02.306548 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:02.308051 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:02.309790 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:02.310490 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:02.310632 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:02.315454 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:02.315463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:02.315480 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:02.315519 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:02.319337 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:02.320137 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:02.320150 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:02.320159 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:02.320389 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:02.320765 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:02.320770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:02.320774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:02.321563 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:02.321886 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:02.322273 (+ 387us) server_negotiation.cc:299] Negotiation successful
0811 12:34:02.322760 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":351,"threads_started":1}
I20260811 12:34:02.327663 14615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } }
I20260811 12:34:02.328061 14615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:02.328189 14615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917d930be6064324b7e8f465196a501e, State: Initialized, Role: FOLLOWER
I20260811 12:34:02.328799 14615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } }
I20260811 12:34:02.329026 14615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:34:02.329156 14615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:34:02.329309 14615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:02.331542 14615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } }
I20260811 12:34:02.332156 14615 leader_election.cc:302] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 917d930be6064324b7e8f465196a501e; no voters:
I20260811 12:34:02.332798 14615 leader_election.cc:288] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:34:02.332935 14623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:02.333894 14623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 1 LEADER]: Becoming Leader. State: Replica: 917d930be6064324b7e8f465196a501e, State: Running, Role: LEADER
I20260811 12:34:02.334455 14623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } }
I20260811 12:34:02.334789 14615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:02.337190 14625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } }
I20260811 12:34:02.337277 14626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 917d930be6064324b7e8f465196a501e. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } }
I20260811 12:34:02.337564 14626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:02.337564 14625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:02.338521 14637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:34:02.344560 14637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:34:02.354784 14637 catalog_manager.cc:1347] Generated new cluster ID: 705d5cb53759433c9dfe5c8be7949bb0
I20260811 12:34:02.354924 14637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:34:02.366918 14637 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:34:02.368506 14637 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:34:02.379066 14637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: Generated new TSK 0
I20260811 12:34:02.380237 14637 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab.
Entry for principal oryx/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab.
I20260811 12:34:02.447439 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36811
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab
--principal=oryx/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:02.785166 14662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:02.785493 14662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:02.785615 14662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:02.793092 14662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:02.793208 14662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:02.793359 14662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:34:02.803398 14662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab
--principal=oryx/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:02.804802 14662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:02.806151 14662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:02.819849 14668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:02.819833 14674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:02.819872 14672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:02.821039 14662 server_base.cc:1034] running on GCE node
I20260811 12:34:02.821718 14662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:02.822832 14662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:02.824155 14662 hybrid_clock.cc:648] HybridClock initialized: now 1786451642824099 us; error 48 us; skew 500 ppm
I20260811 12:34:02.824472 14662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:02 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451642, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:02.827723 14662 init.cc:377] Logged in from keytab as oryx/127.3.204.65@KRBTEST.COM (short username oryx)
I20260811 12:34:02.829299 14662 webserver.cc:460] Webserver started at http://127.3.204.65:44175/ using document root <none> and password file <none>
I20260811 12:34:02.830016 14662 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:02.830138 14662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:02.830441 14662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:02.833585 14662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "9db67396c57e416ca0503984086065d0"
format_stamp: "Formatted at 2026-08-11 12:34:02 on dist-test-slave-lgr5"
server_key: "49c3a30b2780df238013643e19771de1"
server_key_iv: "11f21c29817b8c119dcbff69fdc1f135"
server_key_version: "encryptionkey@0"
I20260811 12:34:02.834362 14662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "9db67396c57e416ca0503984086065d0"
format_stamp: "Formatted at 2026-08-11 12:34:02 on dist-test-slave-lgr5"
server_key: "49c3a30b2780df238013643e19771de1"
server_key_iv: "11f21c29817b8c119dcbff69fdc1f135"
server_key_version: "encryptionkey@0"
I20260811 12:34:02.840176 14662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 12:34:02.844112 14686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:02.845667 14662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:34:02.845847 14662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "9db67396c57e416ca0503984086065d0"
format_stamp: "Formatted at 2026-08-11 12:34:02 on dist-test-slave-lgr5"
server_key: "49c3a30b2780df238013643e19771de1"
server_key_iv: "11f21c29817b8c119dcbff69fdc1f135"
server_key_version: "encryptionkey@0"
I20260811 12:34:02.846068 14662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:02.876471 14662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:02.880981 14662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:02.881266 14662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:02.882232 14662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:02.883632 14662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:02.883708 14662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:02.883826 14662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:02.883884 14662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:02.914044 14662 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:34057
I20260811 12:34:02.914112 14884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:34057 every 8 connection(s)
I20260811 12:34:02.915340 14662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:34:02.924142 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 14662
I20260811 12:34:02.924335 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:34:02.924968 3889 external_mini_cluster.cc:1428] Setting key 63e989210daaf509aa394e14335d37cb
Aug 11 12:34:02 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451642, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:02.936128 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:02.917922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:39475 (local address 127.3.204.126:36811)
0811 12:34:02.918354 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:02.918358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:02.919284 (+ 926us) server_negotiation.cc:407] Connection header received
0811 12:34:02.921168 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:02.921175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:02.921255 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:02.921351 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:02.922921 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:02.923684 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:02.925605 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:02.926058 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:02.926193 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:02.929939 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:02.929945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:02.929948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:02.929986 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:02.932692 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:02.933509 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:02.933514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:02.933517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:02.933653 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:02.934140 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:02.934144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:02.934147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:02.934452 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:02.934598 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:02.935598 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0811 12:34:02.935815 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":322,"thread_start_us":212,"threads_started":1}
I20260811 12:34:02.937278 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:02.918397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.65:39475)
0811 12:34:02.919125 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:02.919164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:02.920884 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:02.921613 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:02.921622 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:02.922144 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:02.922744 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:02.922762 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:02.923890 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:02.923895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:02.925419 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:02.925434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:02.926320 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:02.926329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:02.926603 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:02.927306 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:02.927332 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:02.929683 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:02.932873 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:02.932883 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:02.932901 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:02.933365 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:02.933777 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:02.933783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:02.933787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:02.934008 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:02.934625 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:02.934634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:02.935125 (+ 491us) client_negotiation.cc:769] Sending connection context
0811 12:34:02.935441 (+ 316us) client_negotiation.cc:240] Negotiation successful
0811 12:34:02.936037 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":274,"threads_started":1}
I20260811 12:34:02.938841 14885 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36811
I20260811 12:34:02.939162 14885 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:02.939975 14885 heartbeater.cc:511] Master 127.3.204.126:36811 requested a full tablet report, sending...
I20260811 12:34:02.942431 14563 ts_manager.cc:194] Registered new tserver with Master: 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057)
I20260811 12:34:02.944582 14563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:39475
WARNING: no policy specified for oryx/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab.
Entry for principal oryx/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab.
I20260811 12:34:02.979478 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36811
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab
--principal=oryx/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:03.314605 14894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:03.314921 14894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:03.315076 14894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:03.322427 14894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:03.322554 14894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:03.322710 14894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:34:03.332708 14894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab
--principal=oryx/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:03.334259 14894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:03.335826 14894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:03.349493 14905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:03.349526 14903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:03.349818 14901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:03.350448 14894 server_base.cc:1034] running on GCE node
I20260811 12:34:03.351141 14894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:03.352159 14894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:03.353359 14894 hybrid_clock.cc:648] HybridClock initialized: now 1786451643353295 us; error 56 us; skew 500 ppm
I20260811 12:34:03.353689 14894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:03 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451643, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:03.360142 14894 init.cc:377] Logged in from keytab as oryx/127.3.204.66@KRBTEST.COM (short username oryx)
I20260811 12:34:03.361863 14894 webserver.cc:460] Webserver started at http://127.3.204.66:34073/ using document root <none> and password file <none>
I20260811 12:34:03.362635 14894 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:03.362759 14894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:03.363103 14894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:03.366427 14894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "9000d312ec7242a4902ab42e3d31a271"
format_stamp: "Formatted at 2026-08-11 12:34:03 on dist-test-slave-lgr5"
server_key: "41f94e0e7a5177fd7caebf66b2d899b1"
server_key_iv: "94d314a70c5a404c8aafda3734c5560a"
server_key_version: "encryptionkey@0"
I20260811 12:34:03.367249 14894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "9000d312ec7242a4902ab42e3d31a271"
format_stamp: "Formatted at 2026-08-11 12:34:03 on dist-test-slave-lgr5"
server_key: "41f94e0e7a5177fd7caebf66b2d899b1"
server_key_iv: "94d314a70c5a404c8aafda3734c5560a"
server_key_version: "encryptionkey@0"
I20260811 12:34:03.372767 14894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:34:03.376564 14915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:03.378005 14894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260811 12:34:03.378165 14894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "9000d312ec7242a4902ab42e3d31a271"
format_stamp: "Formatted at 2026-08-11 12:34:03 on dist-test-slave-lgr5"
server_key: "41f94e0e7a5177fd7caebf66b2d899b1"
server_key_iv: "94d314a70c5a404c8aafda3734c5560a"
server_key_version: "encryptionkey@0"
I20260811 12:34:03.378368 14894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:03.401937 14894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:03.406786 14894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:03.407091 14894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:03.408172 14894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:03.409641 14894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:03.409721 14894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:03.409847 14894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:03.409899 14894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:03.441390 14894 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:40033
I20260811 12:34:03.441447 15117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:40033 every 8 connection(s)
I20260811 12:34:03.442749 14894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:34:03.443373 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 14894
I20260811 12:34:03.443557 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:34:03.444221 3889 external_mini_cluster.cc:1428] Setting key 6bd36424507b5dd75684954c98f2b39b
Aug 11 12:34:03 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451643, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:03.466387 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:03.445792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:32853 (local address 127.3.204.126:36811)
0811 12:34:03.446285 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:03.446292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:03.447496 (+ 1204us) server_negotiation.cc:407] Connection header received
0811 12:34:03.450094 (+ 2598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:03.450099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:03.450175 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:03.450279 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:03.452343 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:03.453120 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:03.455195 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:03.455668 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:03.455778 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:03.459505 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:03.459512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:03.459516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:03.459557 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:03.462923 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:03.463729 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:03.463734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:03.463737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:03.463908 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:03.464403 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:03.464408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:03.464411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:03.464799 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:03.464951 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:03.465735 (+ 784us) server_negotiation.cc:299] Negotiation successful
0811 12:34:03.465975 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":244,"threads_started":1}
I20260811 12:34:03.467312 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:03.446380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.66:32853)
0811 12:34:03.447283 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:03.447356 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:03.449716 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:03.450550 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:03.450563 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:03.451299 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:03.452122 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:03.452149 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:03.453290 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:03.453295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:03.455023 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:03.455038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:03.455908 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:03.455915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:03.456157 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:03.456857 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:03.456881 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:03.459217 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:03.463088 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:03.463097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:03.463112 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:03.463585 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:03.464004 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:03.464007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:03.464009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:03.464262 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:03.464968 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:03.464976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:03.465342 (+ 366us) client_negotiation.cc:769] Sending connection context
0811 12:34:03.465595 (+ 253us) client_negotiation.cc:240] Negotiation successful
0811 12:34:03.466041 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":290,"threads_started":1}
I20260811 12:34:03.468987 15118 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36811
I20260811 12:34:03.469309 15118 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:03.470078 15118 heartbeater.cc:511] Master 127.3.204.126:36811 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.204.67@KRBTEST.COM; defaulting to no policy
I20260811 12:34:03.471725 14563 ts_manager.cc:194] Registered new tserver with Master: 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033)
I20260811 12:34:03.472710 14563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:32853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab.
Entry for principal oryx/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab.
I20260811 12:34:03.502990 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36811
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab
--principal=oryx/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:03.836787 15126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:03.837101 15126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:03.837226 15126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:03.844820 15126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:03.844938 15126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:03.845094 15126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:34:03.855031 15126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab
--principal=oryx/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:36811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:03.856395 15126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:03.857714 15126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:03.870536 15141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:03.870590 15144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:03.870621 15139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:03.871176 15126 server_base.cc:1034] running on GCE node
I20260811 12:34:03.871803 15126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:03.872822 15126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:03.874044 15126 hybrid_clock.cc:648] HybridClock initialized: now 1786451643873994 us; error 41 us; skew 500 ppm
I20260811 12:34:03.874368 15126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:03 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451643, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:03.878311 15126 init.cc:377] Logged in from keytab as oryx/127.3.204.67@KRBTEST.COM (short username oryx)
I20260811 12:34:03.879885 15126 webserver.cc:460] Webserver started at http://127.3.204.67:40941/ using document root <none> and password file <none>
I20260811 12:34:03.880667 15126 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:03.880795 15126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:03.881110 15126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:03.884135 15126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "cd5b7a24432e46baac22efd6d1d3ca30"
format_stamp: "Formatted at 2026-08-11 12:34:03 on dist-test-slave-lgr5"
server_key: "b18b55735b56e40cb4d9cf693da79745"
server_key_iv: "59fe8f0ae84a284ebea995390cab7b10"
server_key_version: "encryptionkey@0"
I20260811 12:34:03.884902 15126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "cd5b7a24432e46baac22efd6d1d3ca30"
format_stamp: "Formatted at 2026-08-11 12:34:03 on dist-test-slave-lgr5"
server_key: "b18b55735b56e40cb4d9cf693da79745"
server_key_iv: "59fe8f0ae84a284ebea995390cab7b10"
server_key_version: "encryptionkey@0"
I20260811 12:34:03.890106 15126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 12:34:03.893715 15164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:03.895169 15126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260811 12:34:03.895336 15126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "cd5b7a24432e46baac22efd6d1d3ca30"
format_stamp: "Formatted at 2026-08-11 12:34:03 on dist-test-slave-lgr5"
server_key: "b18b55735b56e40cb4d9cf693da79745"
server_key_iv: "59fe8f0ae84a284ebea995390cab7b10"
server_key_version: "encryptionkey@0"
I20260811 12:34:03.895545 15126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:03.909441 15126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:03.913581 15126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:03.913877 15126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:03.914860 15126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:03.916268 15126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:03.916347 15126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:03.916474 15126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:03.916528 15126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:03.947554 15126 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:42597
I20260811 12:34:03.947602 15348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:42597 every 8 connection(s)
I20260811 12:34:03.948558 14885 heartbeater.cc:503] Master 127.3.204.126:36811 was elected leader, sending a full tablet report...
I20260811 12:34:03.949055 15126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:34:03.956564 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 15126
I20260811 12:34:03.956727 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:34:03.957324 3889 external_mini_cluster.cc:1428] Setting key 9ba17f59717cce269ef3e543178dbd6f
Aug 11 12:34:03 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451643, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:03.969179 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:03.951820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:50725 (local address 127.3.204.126:36811)
0811 12:34:03.952015 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:03.952020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:03.953229 (+ 1209us) server_negotiation.cc:407] Connection header received
0811 12:34:03.955038 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:03.955042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:03.955130 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:03.955217 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:03.956858 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:03.957912 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:03.959609 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:03.960282 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:03.960428 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:03.963894 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:03.963899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:03.963902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:03.963938 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:03.966538 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:03.967149 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:03.967153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:03.967155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:03.967264 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:03.967639 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:03.967641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:03.967643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:03.967951 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:03.968124 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:03.968767 (+ 643us) server_negotiation.cc:299] Negotiation successful
0811 12:34:03.968944 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 12:34:03.970113 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:03.952334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.67:50725)
0811 12:34:03.953067 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:03.953107 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:03.954759 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:03.955420 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:03.955428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:03.955968 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:03.956601 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:03.956620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:03.958071 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:03.958077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:03.959485 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:03.959498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:03.960529 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:03.960536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:03.960802 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:03.961488 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:03.961510 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:03.963687 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:03.966643 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:03.966649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:03.966662 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:03.967059 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:03.967359 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:03.967362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:03.967364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:03.967560 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:03.968058 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:03.968063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:03.968409 (+ 346us) client_negotiation.cc:769] Sending connection context
0811 12:34:03.968667 (+ 258us) client_negotiation.cc:240] Negotiation successful
0811 12:34:03.969053 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":268,"threads_started":1}
I20260811 12:34:03.971707 15351 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36811
I20260811 12:34:03.972038 15351 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:03.972952 15351 heartbeater.cc:511] Master 127.3.204.126:36811 requested a full tablet report, sending...
I20260811 12:34:03.974714 14563 ts_manager.cc:194] Registered new tserver with Master: cd5b7a24432e46baac22efd6d1d3ca30 (127.3.204.67:42597)
I20260811 12:34:03.976491 14563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:50725
I20260811 12:34:03.976548 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.204.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.125.keytab.
Entry for principal oryx/127.3.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.125.keytab.
I20260811 12:34:04.031872 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.125:32801
--webserver_interface=127.3.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.125.keytab
--principal=oryx/127.3.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:36811,127.3.204.125:32801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:04.375335 15357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:04.375701 15357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:04.375847 15357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:04.383479 15357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:04.383610 15357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:04.383690 15357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:04.383771 15357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:04.383834 15357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:04.394361 15357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:36811,127.3.204.125:32801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.125.keytab
--principal=oryx/127.3.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.125:32801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
W20260811 12:34:04.394788 15357 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.204.126:36811,127.3.204.125:32801'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260811 12:34:04.395895 15357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:04.397452 15357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:04.409389 15365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:04.409317 15366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:04.409350 15369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:04.410441 15357 server_base.cc:1034] running on GCE node
I20260811 12:34:04.411257 15357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:04.412741 15357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:04.413971 15357 hybrid_clock.cc:648] HybridClock initialized: now 1786451644413888 us; error 49 us; skew 500 ppm
I20260811 12:34:04.414297 15357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:04 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:04.417398 15357 init.cc:377] Logged in from keytab as oryx/127.3.204.125@KRBTEST.COM (short username oryx)
I20260811 12:34:04.418919 15357 webserver.cc:460] Webserver started at http://127.3.204.125:32787/ using document root <none> and password file <none>
I20260811 12:34:04.419620 15357 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:04.419726 15357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:04.420053 15357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:04.423127 15357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data/instance:
uuid: "957dd776a08d4c5cab22af13198114e5"
format_stamp: "Formatted at 2026-08-11 12:34:04 on dist-test-slave-lgr5"
server_key: "37b0c7545c1f257855ab76e5b028ec2b"
server_key_iv: "5c9947b68fee33f219b6ee0b37d13c51"
server_key_version: "encryptionkey@0"
I20260811 12:34:04.423913 15357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal/instance:
uuid: "957dd776a08d4c5cab22af13198114e5"
format_stamp: "Formatted at 2026-08-11 12:34:04 on dist-test-slave-lgr5"
server_key: "37b0c7545c1f257855ab76e5b028ec2b"
server_key_iv: "5c9947b68fee33f219b6ee0b37d13c51"
server_key_version: "encryptionkey@0"
I20260811 12:34:04.429380 15357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 12:34:04.432972 15377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:04.434350 15357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:34:04.434527 15357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
uuid: "957dd776a08d4c5cab22af13198114e5"
format_stamp: "Formatted at 2026-08-11 12:34:04 on dist-test-slave-lgr5"
server_key: "37b0c7545c1f257855ab76e5b028ec2b"
server_key_iv: "5c9947b68fee33f219b6ee0b37d13c51"
server_key_version: "encryptionkey@0"
I20260811 12:34:04.434747 15357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:04.456013 15357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:04.460412 15357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:04.460672 15357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:04.476061 15118 heartbeater.cc:503] Master 127.3.204.126:36811 was elected leader, sending a full tablet report...
I20260811 12:34:04.480037 15357 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.125:32801
I20260811 12:34:04.480047 15454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.125:32801 every 8 connection(s)
I20260811 12:34:04.481479 15357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
I20260811 12:34:04.485605 15455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:04.485957 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 15357
I20260811 12:34:04.496037 15455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:34:04 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.125@KRBTEST.COM
Aug 11 12:34:04 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.125@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:04.511332 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.488259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.3.204.125:32801)
0811 12:34:04.489125 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:04.489140 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:04.489207 (+ 67us) server_negotiation.cc:407] Connection header received
0811 12:34:04.490912 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:04.490951 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:04.491553 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:04.492059 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:04.493554 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.495462 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.496909 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.497614 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.497802 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.503020 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:04.503033 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:04.503047 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:04.503101 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:04.506881 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.507798 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.507806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.507814 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.508031 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.508446 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.508451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.508453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.509016 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:04.509326 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:04.509797 (+ 471us) server_negotiation.cc:299] Negotiation successful
0811 12:34:04.510207 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":335,"threads_started":1}
I20260811 12:34:04.514233 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.498689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39024)
0811 12:34:04.499476 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:04.499519 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:04.499674 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:04.500192 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:04.500198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:04.500909 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:04.501166 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.501175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.502336 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.502341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.503826 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.503840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.504721 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.504729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.504885 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.505534 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:04.505562 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:04.508311 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:04.511485 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.511496 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.511504 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.512116 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.512465 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.512469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.512473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.512643 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.513334 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:04.513340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:04.513531 (+ 191us) client_negotiation.cc:769] Sending connection context
0811 12:34:04.513684 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 12:34:04.513936 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":379,"threads_started":1}
I20260811 12:34:04.514423 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.498688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.3.204.126:36811)
0811 12:34:04.499172 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:04.499178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:04.499598 (+ 420us) server_negotiation.cc:407] Connection header received
0811 12:34:04.499805 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:04.499811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:04.499915 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:04.500042 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:04.501329 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.502186 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.503989 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.504514 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.504614 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.508504 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:04.508510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:04.508513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:04.508547 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:04.511345 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.512228 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.512232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.512234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.512358 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.512742 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.512746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.512749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.513206 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:04.513359 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:04.513838 (+ 479us) server_negotiation.cc:299] Negotiation successful
0811 12:34:04.514076 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":249,"threads_started":1}
I20260811 12:34:04.516234 15455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:34:04 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.125@KRBTEST.COM for oryx/127.3.204.125@KRBTEST.COM
I20260811 12:34:04.527917 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.517053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.0.0.1:46502)
0811 12:34:04.517239 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:04.517257 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:04.517346 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:04.517580 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:04.517583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:04.517898 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:04.518044 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.518052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.518994 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.518999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.519887 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.519899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.520624 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.520629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.520762 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.521283 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:04.521301 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:04.523265 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:04.525981 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.525985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.525988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.526351 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.526633 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.526636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.526638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.526754 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.527225 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:04.527228 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:04.527354 (+ 126us) client_negotiation.cc:769] Sending connection context
0811 12:34:04.527459 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 12:34:04.527671 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 12:34:04.528182 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.517066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.3.204.125:32801)
0811 12:34:04.517213 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:04.517217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:04.517306 (+ 89us) server_negotiation.cc:407] Connection header received
0811 12:34:04.517418 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:04.517421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:04.517482 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:04.517557 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:04.518154 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.518836 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.520011 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.520450 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.520553 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.523440 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:04.523446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:04.523451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:04.523482 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:04.525871 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.526443 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.526446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.526449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.526557 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.526836 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.526839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.526841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.527132 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:04.527237 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:04.527540 (+ 303us) server_negotiation.cc:299] Negotiation successful
0811 12:34:04.527793 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":47}
I20260811 12:34:04.533004 15455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Bootstrap starting.
I20260811 12:34:04.536324 15455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:04.537395 15455 log.cc:824] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:04.540489 15455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: No bootstrap required, opened a new log
I20260811 12:34:04.547798 15455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } }
I20260811 12:34:04.548189 15455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:04.548301 15455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957dd776a08d4c5cab22af13198114e5, State: Initialized, Role: FOLLOWER
I20260811 12:34:04.548904 15455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } }
I20260811 12:34:04.549779 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } } }
I20260811 12:34:04.550150 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:34:04.550861 15455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 12:34:04.559393 15484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 12:34:04.559516 15484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 12:34:04.563755 15357 master_runner.cc:428] Detected that this master 957dd776a08d4c5cab22af13198114e5 is joining an existing cluster
I20260811 12:34:04.563887 15357 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.204.125:32801
I20260811 12:34:04.593823 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.568162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.3.204.125:32801)
0811 12:34:04.568383 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:04.568393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:04.578401 (+ 10008us) server_negotiation.cc:407] Connection header received
0811 12:34:04.578560 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:04.578565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:04.578635 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:04.578695 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:04.580312 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.581412 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.583536 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.584269 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.584363 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.586843 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:04.586850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:04.586854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:04.586889 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:04.590475 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.591416 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.591422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.591425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.591592 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.592058 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.592063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.592066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.592484 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:04.592632 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:04.593176 (+ 544us) server_negotiation.cc:299] Negotiation successful
0811 12:34:04.593424 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":52}
I20260811 12:34:04.593914 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.568715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.3.204.126:36811)
0811 12:34:04.568904 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:04.568910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:04.577890 (+ 8980us) server_negotiation.cc:407] Connection header received
0811 12:34:04.578258 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:04.578265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:04.578376 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:04.578477 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:04.580116 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.581139 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.583246 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.583832 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.583923 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.586869 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:04.586875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:04.586879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:04.586918 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:04.590482 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.591413 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.591419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.591422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.591592 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.592144 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.592149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.592153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.592569 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:04.592692 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:04.593314 (+ 622us) server_negotiation.cc:299] Negotiation successful
0811 12:34:04.593574 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 12:34:04.595042 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.568543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.0.0.1:46516)
0811 12:34:04.578269 (+ 9726us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:04.578288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:04.578434 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:04.578844 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:04.578846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:04.579756 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:04.580155 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.580169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.581652 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.581660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.583393 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.583408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.584559 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.584570 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.584732 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.585343 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:04.585367 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:04.586651 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:04.590624 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.590629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.590634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.591300 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.591732 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.591736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.591740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.591938 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.592723 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:04.592727 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:04.592878 (+ 151us) client_negotiation.cc:769] Sending connection context
0811 12:34:04.594237 (+ 1359us) client_negotiation.cc:240] Negotiation successful
0811 12:34:04.594550 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9596,"thread_start_us":9385,"threads_started":1}
I20260811 12:34:04.595711 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.568195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39030)
0811 12:34:04.577742 (+ 9547us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:04.577773 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:04.578039 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:04.578512 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:04.578516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:04.579432 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:04.579862 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.579879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.581314 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.581319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.583077 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.583095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.584233 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.584238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.584648 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.585462 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:04.585483 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:04.586658 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:04.590653 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.590657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.590661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.591274 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.591749 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.591754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.591758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.591971 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.592800 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:04.592804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:04.593016 (+ 212us) client_negotiation.cc:769] Sending connection context
0811 12:34:04.594558 (+ 1542us) client_negotiation.cc:240] Negotiation successful
0811 12:34:04.594893 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9376,"mutex_wait_us":26,"thread_start_us":9191,"threads_started":1}
WARNING: no policy specified for oryx/127.3.204.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.124@KRBTEST.COM" created.
Aug 11 12:34:04 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451642, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.126@KRBTEST.COM for oryx/127.3.204.125@KRBTEST.COM
I20260811 12:34:04.627242 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.602937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.3.204.125:32801)
0811 12:34:04.603119 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:04.603125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:04.609213 (+ 6088us) server_negotiation.cc:407] Connection header received
0811 12:34:04.609681 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:04.609687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:04.609813 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:04.609942 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:04.611533 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.612754 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.614680 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.615412 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.615552 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.619918 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:04.619925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:04.619930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:04.619978 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:04.623877 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.624904 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.624909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.624912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.625064 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.625501 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.625505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.625507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.625910 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:04.626039 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:04.626649 (+ 610us) server_negotiation.cc:299] Negotiation successful
0811 12:34:04.626896 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 12:34:04.627311 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.602916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.0.0.1:46528)
0811 12:34:04.609026 (+ 6110us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:04.609085 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:04.609393 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:04.610010 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:04.610017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:04.610949 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:04.611354 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.611368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.612928 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.612934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.614510 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.614529 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.615610 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.615616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.615810 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.616665 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:04.616689 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:04.619670 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:04.624055 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.624072 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.624113 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.624765 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.625181 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.625186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.625190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.625387 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.626050 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:04.626059 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:04.626277 (+ 218us) client_negotiation.cc:769] Sending connection context
0811 12:34:04.626490 (+ 213us) client_negotiation.cc:240] Negotiation successful
0811 12:34:04.626863 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5783,"thread_start_us":5590,"threads_started":1}
I20260811 12:34:04.629544 14563 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.204.125:32801: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "957dd776a08d4c5cab22af13198114e5"
member_type: NON_VOTER
last_known_addr {
host: "127.3.204.125"
port: 32801
}
attrs {
promote: true
}
}
dest_uuid: "917d930be6064324b7e8f465196a501e"
cas_config_opid_index: -1
I20260811 12:34:04.630988 14563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.124.keytab.
Entry for principal oryx/127.3.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.124.keytab.
I20260811 12:34:04.634289 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.124:39971
--webserver_interface=127.3.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.124.keytab
--principal=oryx/127.3.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:36811,127.3.204.125:32801,127.3.204.124:39971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:34:04.643577 15506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 957dd776a08d4c5cab22af13198114e5 (127.3.204.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } }
I20260811 12:34:04.644706 15414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:04.648622 15414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Refusing update from remote peer 917d930be6064324b7e8f465196a501e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260811 12:34:04.648671 15506 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.204.125:32801
I20260811 12:34:04.649978 15357 master.cc:537] Master@127.3.204.125:32801 shutting down...
I20260811 12:34:04.651305 15506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Connected to new peer: Peer: permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:04.651937 15507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } }
I20260811 12:34:04.652259 15507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
W20260811 12:34:04.663651 14538 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260811 12:34:04.668504 15357 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:04.668982 15357 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:04.669199 15357 tablet_replica.cc:330] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: stopping tablet replica
W20260811 12:34:04.669322 14538 consensus_peers.cc:487] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e -> Peer 957dd776a08d4c5cab22af13198114e5 (127.3.204.125:32801): Couldn't send request to peer 957dd776a08d4c5cab22af13198114e5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 12:34:04.675509 15357 master.cc:552] Master@127.3.204.125:32801 shutdown complete.
I20260811 12:34:04.675639 15357 master_runner.cc:305] Clearing existing system tablet
I20260811 12:34:04.676051 15357 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:04.676167 15357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:04.678402 15357 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260811 12:34:04.679842 15516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:04.680178 15357 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:04.680418 15357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 12:34:04.680600 15357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
uuid: "957dd776a08d4c5cab22af13198114e5"
format_stamp: "Formatted at 2026-08-11 12:34:04 on dist-test-slave-lgr5"
server_key: "37b0c7545c1f257855ab76e5b028ec2b"
server_key_iv: "5c9947b68fee33f219b6ee0b37d13c51"
server_key_version: "encryptionkey@0"
I20260811 12:34:04.684580 15357 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:04.689278 15357 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 12:34:04.689447 15357 log.cc:1196] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260811 12:34:04.689845 15357 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Deleting consensus metadata
I20260811 12:34:04.690086 15357 master_runner.cc:315] Copying system tablet from 127.3.204.126:36811
I20260811 12:34:04.692819 15357 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: tablet copy: Beginning tablet copy session from remote peer at address 127.3.204.126:36811
I20260811 12:34:04.705399 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.693765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.3.204.126:36811)
0811 12:34:04.693984 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:04.693992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:04.694458 (+ 466us) server_negotiation.cc:407] Connection header received
0811 12:34:04.694652 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:04.694657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:04.694725 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:04.694807 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:04.695702 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.696508 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.697750 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.698197 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.698307 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.700232 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:04.700239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:04.700244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:04.700283 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:04.702822 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.703415 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.703421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.703424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.703573 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:04.703927 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:04.703930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:04.703932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:04.704256 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:04.704400 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:04.704779 (+ 379us) server_negotiation.cc:299] Negotiation successful
0811 12:34:04.704995 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 12:34:04.705435 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:04.693712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39042)
0811 12:34:04.694333 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:04.694353 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:04.694498 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:04.694854 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:04.694859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:04.695338 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:04.695555 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.695565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.696642 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.696647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.697599 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:04.697611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:04.698343 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:04.698348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:04.698537 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:04.699192 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:04.699210 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:04.699904 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:04.702946 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.702960 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.702964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.703308 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.703681 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:04.703690 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:04.703694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:04.703826 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:04.704374 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:04.704380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:04.704544 (+ 164us) client_negotiation.cc:769] Sending connection context
0811 12:34:04.704666 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 12:34:04.705002 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":400,"threads_started":1}
I20260811 12:34:04.706687 14589 tablet_copy_service.cc:140] P 917d930be6064324b7e8f465196a501e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 957dd776a08d4c5cab22af13198114e5 ({username='oryx', principal='oryx/127.3.204.125@KRBTEST.COM'} at 127.0.0.1:39042)
I20260811 12:34:04.707064 14589 tablet_copy_service.cc:161] P 917d930be6064324b7e8f465196a501e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 957dd776a08d4c5cab22af13198114e5 at {username='oryx', principal='oryx/127.3.204.125@KRBTEST.COM'} at 127.0.0.1:39042: session id = 957dd776a08d4c5cab22af13198114e5-00000000000000000000000000000000
I20260811 12:34:04.710568 14589 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 12:34:04.713093 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 12:34:04.718071 15357 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: tablet copy: Starting download of 0 data blocks...
I20260811 12:34:04.718425 15357 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: tablet copy: Starting download of 1 WAL segments...
I20260811 12:34:04.721882 15357 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 12:34:04.725543 14589 tablet_copy_service.cc:342] P 917d930be6064324b7e8f465196a501e: Request end of tablet copy session 957dd776a08d4c5cab22af13198114e5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.204.125@KRBTEST.COM'} at 127.0.0.1:39042
I20260811 12:34:04.725804 14589 tablet_copy_service.cc:433] P 917d930be6064324b7e8f465196a501e: ending tablet copy session 957dd776a08d4c5cab22af13198114e5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 957dd776a08d4c5cab22af13198114e5
I20260811 12:34:04.737171 15357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:04.739831 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:04.745954 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:04.750190 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:04.753122 15357 server_base.cc:1034] running on GCE node
I20260811 12:34:04.753476 15357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:04.754106 15357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:04.755318 15357 hybrid_clock.cc:648] HybridClock initialized: now 1786451644755263 us; error 34 us; skew 500 ppm
I20260811 12:34:04.755534 15357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:04 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:04.757404 15357 init.cc:377] Logged in from keytab as oryx/127.3.204.125@KRBTEST.COM (short username oryx)
I20260811 12:34:04.758587 15357 webserver.cc:460] Webserver started at http://127.3.204.125:35681/ using document root <none> and password file <none>
I20260811 12:34:04.758888 15357 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:04.758957 15357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:04.761364 15357 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260811 12:34:04.763087 15542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:04.763522 15357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 12:34:04.763654 15357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
uuid: "957dd776a08d4c5cab22af13198114e5"
format_stamp: "Formatted at 2026-08-11 12:34:04 on dist-test-slave-lgr5"
server_key: "37b0c7545c1f257855ab76e5b028ec2b"
server_key_iv: "5c9947b68fee33f219b6ee0b37d13c51"
server_key_version: "encryptionkey@0"
I20260811 12:34:04.763792 15357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:04.779685 15357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:04.783281 15357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:04.814479 15357 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.125:32801
I20260811 12:34:04.814483 15631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.125:32801 every 8 connection(s)
I20260811 12:34:04.815399 15357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
I20260811 12:34:04.821483 15632 sys_catalog.cc:263] Verifying existing consensus state
I20260811 12:34:04.824730 15632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Bootstrap starting.
I20260811 12:34:04.839426 15632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:34:04.839895 15632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Bootstrap complete.
I20260811 12:34:04.841010 15632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } }
I20260811 12:34:04.841292 15632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 957dd776a08d4c5cab22af13198114e5, State: Initialized, Role: LEARNER
I20260811 12:34:04.841559 15632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } }
I20260811 12:34:04.842411 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } }
I20260811 12:34:04.842723 15632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:04.842844 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: This master's current role is: LEARNER
I20260811 12:34:04.851739 15661 catalog_manager.cc:1259] Loaded cluster ID: 705d5cb53759433c9dfe5c8be7949bb0
I20260811 12:34:04.851920 15661 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: loading cluster ID for follower catalog manager: success
I20260811 12:34:04.855497 15661 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: acquiring CA information for follower catalog manager: success
I20260811 12:34:04.858271 15661 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 12:34:04.980000 15351 heartbeater.cc:503] Master 127.3.204.126:36811 was elected leader, sending a full tablet report...
W20260811 12:34:05.056772 15508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:05.057099 15508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:05.057256 15508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:05.064935 15508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:05.065070 15508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:05.065155 15508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:05.065234 15508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:05.065300 15508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:05.076046 15508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:36811,127.3.204.125:32801,127.3.204.124:39971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.124.keytab
--principal=oryx/127.3.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.124:39971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:05.077539 15508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:05.078922 15508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:05.091053 15671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:05.090989 15672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:05.091094 15675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:05.091616 15508 server_base.cc:1034] running on GCE node
I20260811 12:34:05.092561 15508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:05.094081 15508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:05.095306 15508 hybrid_clock.cc:648] HybridClock initialized: now 1786451645095254 us; error 40 us; skew 500 ppm
I20260811 12:34:05.095708 15508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:05.099002 15508 init.cc:377] Logged in from keytab as oryx/127.3.204.124@KRBTEST.COM (short username oryx)
I20260811 12:34:05.100690 15508 webserver.cc:460] Webserver started at http://127.3.204.124:37847/ using document root <none> and password file <none>
I20260811 12:34:05.101399 15508 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:05.101523 15508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:05.101819 15508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:05.105036 15508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data/instance:
uuid: "ccdba4578a1943138c8d21cd870a3978"
format_stamp: "Formatted at 2026-08-11 12:34:05 on dist-test-slave-lgr5"
server_key: "bfb196db11993494bb46dd7ca38ef280"
server_key_iv: "e9eb61a5be798f317bf0b6ca6f1548ee"
server_key_version: "encryptionkey@0"
I20260811 12:34:05.105818 15508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal/instance:
uuid: "ccdba4578a1943138c8d21cd870a3978"
format_stamp: "Formatted at 2026-08-11 12:34:05 on dist-test-slave-lgr5"
server_key: "bfb196db11993494bb46dd7ca38ef280"
server_key_iv: "e9eb61a5be798f317bf0b6ca6f1548ee"
server_key_version: "encryptionkey@0"
I20260811 12:34:05.111507 15508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:34:05.115324 15685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:05.116703 15508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:34:05.116859 15508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
uuid: "ccdba4578a1943138c8d21cd870a3978"
format_stamp: "Formatted at 2026-08-11 12:34:05 on dist-test-slave-lgr5"
server_key: "bfb196db11993494bb46dd7ca38ef280"
server_key_iv: "e9eb61a5be798f317bf0b6ca6f1548ee"
server_key_version: "encryptionkey@0"
I20260811 12:34:05.117067 15508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:05.139389 15508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:05.144073 15508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:05.144418 15508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:05.164919 15508 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.124:39971
I20260811 12:34:05.164934 15783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.124:39971 every 8 connection(s)
I20260811 12:34:05.166436 15508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
I20260811 12:34:05.169986 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 15508
I20260811 12:34:05.170487 15787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:05.181298 15787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.124@KRBTEST.COM
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.124@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:05.197489 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.172318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.3.204.124:39971)
0811 12:34:05.173175 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.173192 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.173249 (+ 57us) server_negotiation.cc:407] Connection header received
0811 12:34:05.175640 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.175673 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.176310 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.176823 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.178235 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.180353 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.182053 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.183121 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.183306 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.188945 (+ 5639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.188956 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.188969 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.189025 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.192226 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.193055 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.193062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.193073 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.193260 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.193865 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.193870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.193874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.194606 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.194889 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.195360 (+ 471us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.195972 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":348,"threads_started":1}
I20260811 12:34:05.200282 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.184545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39048)
0811 12:34:05.185287 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.185339 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.185520 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.185903 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.185912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.186643 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:05.186940 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.186950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.188446 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.188450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.189688 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.189701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.190469 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.190475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.190674 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.191281 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:05.191308 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:05.193631 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:05.197353 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.197362 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.197368 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.197913 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.198310 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.198313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.198316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.198460 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.199118 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:05.199125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:05.199353 (+ 228us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.199607 (+ 254us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.199914 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":340,"threads_started":1}
I20260811 12:34:05.200320 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.184716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.3.204.126:36811)
0811 12:34:05.184974 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.184980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.185457 (+ 477us) server_negotiation.cc:407] Connection header received
0811 12:34:05.185649 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.185656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.185731 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.185817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.187104 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.188251 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.189847 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.190246 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.190403 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.193815 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.193821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.193825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.193871 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.197201 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.198031 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.198036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.198038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.198181 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.198565 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.198569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.198571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.198947 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.199111 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.199733 (+ 622us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.199968 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 12:34:05.202265 15787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.124@KRBTEST.COM for oryx/127.3.204.125@KRBTEST.COM
I20260811 12:34:05.215628 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.209028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.0.0.1:46544)
0811 12:34:05.209595 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.209618 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.209745 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.210231 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.210235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.210260 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:05.210460 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.210472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.211888 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.211893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.213625 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.213640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.214931 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.214936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.215093 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.215141 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.215209 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.215315 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":258,"threads_started":1}
I20260811 12:34:05.215986 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.209086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.3.204.125:32801)
0811 12:34:05.209888 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.209893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.209916 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:34:05.209977 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.209981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.210070 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.210206 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:05.210622 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.211741 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.213782 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.214755 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.214899 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.215195 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.215255 (+ 60us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.215399 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":530,"threads_started":1}
I20260811 12:34:05.217201 15582 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260811 12:34:05.217672 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.203089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.0.0.1:46538)
0811 12:34:05.203276 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.203292 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.203382 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.204062 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.204065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.204573 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:05.204743 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.204751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.206114 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.206120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.207084 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.207097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.208021 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.208026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.208203 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.208789 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:05.208813 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:05.211530 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:05.214747 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.214752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.214755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.215432 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.215970 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.215974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.215977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.216168 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.216731 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:05.216735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:05.216889 (+ 154us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.217005 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.217324 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 12:34:05.217757 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.203181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.3.204.125:32801)
0811 12:34:05.203797 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.203802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.203819 (+ 17us) server_negotiation.cc:407] Connection header received
0811 12:34:05.203873 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.203877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.203938 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.204032 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.204867 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.205970 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.207231 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.207859 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.207998 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.211735 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.211740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.211744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.211777 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.214604 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.215573 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.215582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.215586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.215838 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.216298 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.216304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.216308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.216608 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.216746 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.217155 (+ 409us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.217374 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":311,"threads_started":1}
I20260811 12:34:05.217959 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 917d930be6064324b7e8f465196a501e. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } }
I20260811 12:34:05.218227 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: This master's current role is: LEARNER
I20260811 12:34:05.218822 15787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.124@KRBTEST.COM for oryx/127.3.204.124@KRBTEST.COM
I20260811 12:34:05.230805 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.219593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:39971 (local address 127.0.0.1:41412)
0811 12:34:05.219736 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.219751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.219832 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.220181 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.220184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.220513 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:05.220657 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.220665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.221718 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.221723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.222579 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.222591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.223302 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.223307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.223492 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.224024 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:05.224040 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:05.226057 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:05.228752 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.228756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.228758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.229109 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.229413 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.229417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.229419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.229586 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.230037 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:05.230040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:05.230180 (+ 140us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.230285 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.230499 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 12:34:05.230883 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.219759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.3.204.124:39971)
0811 12:34:05.219906 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.219910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.219925 (+ 15us) server_negotiation.cc:407] Connection header received
0811 12:34:05.219962 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.219966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.220027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.220136 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.220770 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.221540 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.222727 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.223141 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.223245 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.226219 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.226224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.226227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.226257 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.228638 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.229202 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.229206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.229208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.229319 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.229669 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.229672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.229674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.229938 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.230056 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.230370 (+ 314us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.230630 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":50}
I20260811 12:34:05.235832 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Bootstrap starting.
I20260811 12:34:05.239073 15787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:05.240289 15787 log.cc:824] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:05.243353 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: No bootstrap required, opened a new log
I20260811 12:34:05.250874 15787 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } }
I20260811 12:34:05.251267 15787 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:05.251402 15787 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdba4578a1943138c8d21cd870a3978, State: Initialized, Role: FOLLOWER
I20260811 12:34:05.252065 15787 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } }
I20260811 12:34:05.253017 15808 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } } }
I20260811 12:34:05.253340 15808 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:34:05.254063 15787 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 12:34:05.263538 15829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 12:34:05.263688 15829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 12:34:05.267331 15508 master_runner.cc:428] Detected that this master ccdba4578a1943138c8d21cd870a3978 is joining an existing cluster
I20260811 12:34:05.267438 15508 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.204.124:39971
I20260811 12:34:05.290868 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.271686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.3.204.125:32801)
0811 12:34:05.271870 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.271876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.272876 (+ 1000us) server_negotiation.cc:407] Connection header received
0811 12:34:05.273188 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.273195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.273291 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.273380 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.275130 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.276381 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.278678 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.279516 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.279648 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.282868 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.282877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.282882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.282931 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.287063 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.288006 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.288013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.288018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.288239 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.288764 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.288770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.288774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.289181 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.289325 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.290212 (+ 887us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.290488 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:34:05.291906 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.271719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.0.0.1:46560)
0811 12:34:05.272758 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.272782 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.273004 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.273619 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.273624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.274409 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:05.274904 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.274918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.276569 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.276576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.278461 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.278480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.279971 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.279978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.280231 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.281121 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:05.281152 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:05.282595 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:05.287251 (+ 4656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.287258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.287263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.287828 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.288386 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.288391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.288395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.288612 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.289408 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:05.289414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:05.289671 (+ 257us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.289841 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.291390 (+ 1549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":526,"threads_started":1}
I20260811 12:34:05.304404 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 14662
I20260811 12:34:05.305613 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.271744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.3.204.124:39971)
0811 12:34:05.271929 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.271934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.292253 (+ 20319us) server_negotiation.cc:407] Connection header received
0811 12:34:05.292535 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.292540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.292613 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.292676 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.294062 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.295215 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.296796 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.297291 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.297401 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.299378 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.299384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.299387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.299419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.302336 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.303137 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.303142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.303145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.303306 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.303744 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.303749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.303752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.304264 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.304430 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.305013 (+ 583us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.305250 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 12:34:05.306452 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.272863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:39971 (local address 127.0.0.1:41418)
0811 12:34:05.292180 (+ 19317us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.292200 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.292409 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.293119 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.293124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.293666 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:05.293897 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.293910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.295437 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.295441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.296654 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.296668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.297444 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.297449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.297618 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.298195 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:05.298216 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:05.299206 (+ 990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:05.302494 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.302499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.302502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.303014 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.303446 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.303450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.303453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.303639 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.304536 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:05.304540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:05.304697 (+ 157us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.305788 (+ 1091us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.306094 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19195,"thread_start_us":23062,"threads_started":1}
I20260811 12:34:05.317582 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.271310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.3.204.126:36811)
0811 12:34:05.271511 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.271518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.283093 (+ 11575us) server_negotiation.cc:407] Connection header received
0811 12:34:05.283252 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.283258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.283366 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.283474 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.287017 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.288268 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.305362 (+ 17094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.306103 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.306216 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.309921 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.309929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.309934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.309981 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.314153 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.315086 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.315092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.315097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.315288 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.315764 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.315770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.315774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.316230 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.316374 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.316959 (+ 585us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.317187 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 12:34:05.317624 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.272419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39062)
0811 12:34:05.282958 (+ 10539us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.282975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.283125 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.286033 (+ 2908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.286037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.286572 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:05.286839 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.286849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.288475 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.288481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.305151 (+ 16670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.305169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.307001 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.307008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.307196 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.307881 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:05.307906 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:05.309661 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:05.314363 (+ 4702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.314368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.314373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.314904 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.315432 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.315437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.315442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.315632 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.316439 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:05.316443 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:05.316660 (+ 217us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.316781 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.317116 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10386,"thread_start_us":10114,"threads_started":1}
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.65@KRBTEST.COM
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451642, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.126@KRBTEST.COM for oryx/127.3.204.124@KRBTEST.COM
I20260811 12:34:05.336803 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.306308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.3.204.65:34057)
0811 12:34:05.316697 (+ 10389us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.316707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.316741 (+ 34us) server_negotiation.cc:407] Connection header received
0811 12:34:05.316818 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.316829 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.317161 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.319598 (+ 2437us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.319875 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.321512 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.323529 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.324513 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.325448 (+ 935us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.329801 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.329818 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.329839 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.329880 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.333230 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.334028 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.334035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.334037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.334165 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.334667 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.334673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.334677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.335204 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.335653 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.336036 (+ 383us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.336375 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10177,"thread_start_us":9992,"threads_started":1}
I20260811 12:34:05.338495 14878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:05.340224 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.324284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:39971 (local address 127.0.0.1:41426)
0811 12:34:05.324553 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.324572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.324690 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.325080 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.325085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.325556 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:05.325754 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.325767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.327217 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.327223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.328588 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.328605 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.329690 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.329697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.329889 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.330498 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:05.330523 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:05.333399 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:05.337294 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.337300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.337305 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.337730 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.338144 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.338149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.338153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.338335 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.339140 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:05.339146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:05.339376 (+ 230us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.339538 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.339842 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260811 12:34:05.341522 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.324414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.3.204.124:39971)
0811 12:34:05.324580 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.324585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.324621 (+ 36us) server_negotiation.cc:407] Connection header received
0811 12:34:05.324849 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.324854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.324943 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.325026 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.325931 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.327062 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.328764 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.329507 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.329644 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.333626 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.333633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.333642 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.333686 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.337154 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.337856 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.337861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.337863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.337989 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.338491 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.338496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.338503 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.338907 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.339063 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.340739 (+ 1676us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.341002 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:34:05.343410 14563 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.204.124:39971: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ccdba4578a1943138c8d21cd870a3978"
member_type: NON_VOTER
last_known_addr {
host: "127.3.204.124"
port: 39971
}
attrs {
promote: true
}
}
dest_uuid: "917d930be6064324b7e8f465196a501e"
cas_config_opid_index: 5
I20260811 12:34:05.344657 14563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } }
I20260811 12:34:05.347045 15800 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ccdba4578a1943138c8d21cd870a3978 (127.3.204.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.349031 15582 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 LEARNER]: Refusing update from remote peer 917d930be6064324b7e8f465196a501e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 12:34:05.350009 15739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:05.351871 15849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Connected to new peer: Peer: permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260811 12:34:05.353655 15739 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 FOLLOWER]: Refusing update from remote peer 917d930be6064324b7e8f465196a501e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 12:34:05.355145 15582 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ccdba4578a1943138c8d21cd870a3978 (127.3.204.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.356381 15852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:05.356758 15800 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.204.124:39971
I20260811 12:34:05.357657 15508 master.cc:537] Master@127.3.204.124:39971 shutting down...
I20260811 12:34:05.359510 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.359813 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: This master's current role is: LEARNER
I20260811 12:34:05.370857 15849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.371377 15849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:05.378072 15854 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: attempting to promote NON_VOTER 957dd776a08d4c5cab22af13198114e5 to VOTER
I20260811 12:34:05.379189 15854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } }
I20260811 12:34:05.382264 15582 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 LEARNER]: Refusing update from remote peer 917d930be6064324b7e8f465196a501e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260811 12:34:05.382306 15800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: NON_VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: true } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.382679 15800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:05.383090 15800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Connected to new peer: Peer: permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260811 12:34:05.385387 14544 consensus_peers.cc:487] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e -> Peer ccdba4578a1943138c8d21cd870a3978 (127.3.204.124:39971): Couldn't send request to peer ccdba4578a1943138c8d21cd870a3978. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 12:34:05.386085 15854 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 957dd776a08d4c5cab22af13198114e5 (127.3.204.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.386822 15508 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:05.387305 15508 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:05.387537 15508 tablet_replica.cc:330] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: stopping tablet replica
I20260811 12:34:05.390440 15854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.390758 15854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:05.392179 15849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.392443 15849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:05.393702 15582 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 957dd776a08d4c5cab22af13198114e5 (127.3.204.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.396320 15508 master.cc:552] Master@127.3.204.124:39971 shutdown complete.
I20260811 12:34:05.396445 15508 master_runner.cc:305] Clearing existing system tablet
I20260811 12:34:05.396920 15508 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:05.397073 15508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:05.398980 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.399252 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:34:05.401085 15508 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:34:05.402724 15861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:05.403069 15508 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:05.403354 15508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260811 12:34:05.403462 15508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
uuid: "ccdba4578a1943138c8d21cd870a3978"
format_stamp: "Formatted at 2026-08-11 12:34:05 on dist-test-slave-lgr5"
server_key: "bfb196db11993494bb46dd7ca38ef280"
server_key_iv: "e9eb61a5be798f317bf0b6ca6f1548ee"
server_key_version: "encryptionkey@0"
I20260811 12:34:05.408591 15508 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:05.415838 15508 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 12:34:05.416036 15508 log.cc:1196] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260811 12:34:05.416579 15508 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Deleting consensus metadata
I20260811 12:34:05.416846 15508 master_runner.cc:315] Copying system tablet from 127.3.204.126:36811
I20260811 12:34:05.422312 15508 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: tablet copy: Beginning tablet copy session from remote peer at address 127.3.204.126:36811
I20260811 12:34:05.439769 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.423232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.3.204.126:36811)
0811 12:34:05.423489 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.423496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.424685 (+ 1189us) server_negotiation.cc:407] Connection header received
0811 12:34:05.424862 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.424868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.424979 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.425089 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:05.426191 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.427348 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.429293 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.429986 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.430115 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.432589 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:05.432597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:05.432602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:05.432646 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:05.436584 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.437532 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.437538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.437542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.437720 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:05.438167 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:05.438172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:05.438176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:05.438536 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:05.438689 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.439136 (+ 447us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.439390 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260811 12:34:05.439904 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.423154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39074)
0811 12:34:05.424539 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.424568 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.424727 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.425140 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.425145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.425712 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:05.426002 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.426018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.427544 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.427550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.429124 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.429142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.430157 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.430164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.430362 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.431131 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:05.431153 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:05.432349 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:05.436828 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.436833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.436838 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.437393 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.437850 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:05.437855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:05.437859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:05.438046 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:05.438663 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:05.438669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:05.438865 (+ 196us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.439021 (+ 156us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.439374 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"thread_start_us":1105,"threads_started":1}
I20260811 12:34:05.440837 14589 tablet_copy_service.cc:140] P 917d930be6064324b7e8f465196a501e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ccdba4578a1943138c8d21cd870a3978 ({username='oryx', principal='oryx/127.3.204.124@KRBTEST.COM'} at 127.0.0.1:39074)
I20260811 12:34:05.441031 14589 tablet_copy_service.cc:161] P 917d930be6064324b7e8f465196a501e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ccdba4578a1943138c8d21cd870a3978 at {username='oryx', principal='oryx/127.3.204.124@KRBTEST.COM'} at 127.0.0.1:39074: session id = ccdba4578a1943138c8d21cd870a3978-00000000000000000000000000000000
I20260811 12:34:05.444766 14589 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 12:34:05.446914 15508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 12:34:05.454267 15508 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: tablet copy: Starting download of 0 data blocks...
I20260811 12:34:05.454632 15508 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: tablet copy: Starting download of 1 WAL segments...
I20260811 12:34:05.457631 15508 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 12:34:05.463433 14589 tablet_copy_service.cc:342] P 917d930be6064324b7e8f465196a501e: Request end of tablet copy session ccdba4578a1943138c8d21cd870a3978-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.204.124@KRBTEST.COM'} at 127.0.0.1:39074
I20260811 12:34:05.463610 14589 tablet_copy_service.cc:433] P 917d930be6064324b7e8f465196a501e: ending tablet copy session ccdba4578a1943138c8d21cd870a3978-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ccdba4578a1943138c8d21cd870a3978
I20260811 12:34:05.476436 15508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 12:34:05.476553 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 14662
W20260811 12:34:05.491151 15873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:05.492524 15872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:05.500037 15508 server_base.cc:1034] running on GCE node
W20260811 12:34:05.500809 15876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:05.501291 15508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:05.501921 15508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:05.503120 15508 hybrid_clock.cc:648] HybridClock initialized: now 1786451645503072 us; error 41 us; skew 500 ppm
I20260811 12:34:05.503353 15508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:05.505240 15508 init.cc:377] Logged in from keytab as oryx/127.3.204.124@KRBTEST.COM (short username oryx)
I20260811 12:34:05.508987 15508 webserver.cc:460] Webserver started at http://127.3.204.124:36873/ using document root <none> and password file <none>
I20260811 12:34:05.509251 15508 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:05.509353 15508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:05.511650 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:34057
--local_ip_for_outbound_sockets=127.3.204.65
--tserver_master_addrs=127.3.204.126:36811
--webserver_port=44175
--webserver_interface=127.3.204.65
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab
--principal=oryx/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801,127.3.204.124:39971 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:34:05.512308 15508 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260811 12:34:05.514259 15889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:05.514730 15508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260811 12:34:05.514904 15508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
uuid: "ccdba4578a1943138c8d21cd870a3978"
format_stamp: "Formatted at 2026-08-11 12:34:05 on dist-test-slave-lgr5"
server_key: "bfb196db11993494bb46dd7ca38ef280"
server_key_iv: "e9eb61a5be798f317bf0b6ca6f1548ee"
server_key_version: "encryptionkey@0"
I20260811 12:34:05.515165 15508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:05.530766 15508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:05.535058 15508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:05.556203 15508 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.124:39971
I20260811 12:34:05.556247 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.124:39971 every 8 connection(s)
I20260811 12:34:05.557415 15508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
I20260811 12:34:05.562878 15990 sys_catalog.cc:263] Verifying existing consensus state
I20260811 12:34:05.566715 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Bootstrap starting.
I20260811 12:34:05.584156 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:34:05.584759 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: Bootstrap complete.
I20260811 12:34:05.585799 15990 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } }
I20260811 12:34:05.586086 15990 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ccdba4578a1943138c8d21cd870a3978, State: Initialized, Role: LEARNER
I20260811 12:34:05.586344 15990 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } }
I20260811 12:34:05.587103 15993 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.587425 15990 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:05.587579 15993 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: This master's current role is: LEARNER
I20260811 12:34:05.595722 16009 catalog_manager.cc:1259] Loaded cluster ID: 705d5cb53759433c9dfe5c8be7949bb0
I20260811 12:34:05.595822 16009 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: loading cluster ID for follower catalog manager: success
I20260811 12:34:05.598325 16009 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: acquiring CA information for follower catalog manager: success
I20260811 12:34:05.600250 16009 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 12:34:05.927760 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.919584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:39971 (local address 127.0.0.1:41434)
0811 12:34:05.920553 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:05.920570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:05.920694 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:05.921155 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:05.921159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:05.921187 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:05.921381 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.921389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.923220 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.923225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.924917 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:05.924932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.926962 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.926967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:05.927156 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.927185 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 12:34:05.927287 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 12:34:05.927401 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":748,"threads_started":1}
I20260811 12:34:05.928037 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:05.919771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.3.204.124:39971)
0811 12:34:05.920378 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:05.920385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:05.920662 (+ 277us) server_negotiation.cc:407] Connection header received
0811 12:34:05.920830 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:05.920837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:05.920945 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:05.921095 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:05.921542 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.923053 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.925107 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:05.926696 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:05.926877 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:05.927136 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:05.927320 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 12:34:05.927496 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":279,"threads_started":1}
I20260811 12:34:05.928989 15944 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260811 12:34:05.929663 15993 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 917d930be6064324b7e8f465196a501e. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:05.930048 15993 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: This master's current role is: LEARNER
W20260811 12:34:05.934515 15903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:05.934892 15903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:05.935102 15903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:05.945868 15903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:05.946079 15903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:05.946331 15903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:34:05.959681 15903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab
--principal=oryx/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:34057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=44175
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801,127.3.204.124:39971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:05.961292 15903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:05.962850 15903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:05.983868 16023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:05.985858 16025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:05.985967 16027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:05.986227 15903 server_base.cc:1034] running on GCE node
I20260811 12:34:05.987076 15903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:05.988344 15903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:05.989605 15903 hybrid_clock.cc:648] HybridClock initialized: now 1786451645989496 us; error 51 us; skew 500 ppm
I20260811 12:34:05.989995 15903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:05 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:05.994624 15903 init.cc:377] Logged in from keytab as oryx/127.3.204.65@KRBTEST.COM (short username oryx)
I20260811 12:34:05.996907 15903 webserver.cc:460] Webserver started at http://127.3.204.65:44175/ using document root <none> and password file <none>
I20260811 12:34:05.997854 15903 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:05.998003 15903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:06.005831 15903 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 12:34:06.009655 16038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:06.011171 15903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:34:06.011399 15903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "9db67396c57e416ca0503984086065d0"
format_stamp: "Formatted at 2026-08-11 12:34:02 on dist-test-slave-lgr5"
server_key: "49c3a30b2780df238013643e19771de1"
server_key_iv: "11f21c29817b8c119dcbff69fdc1f135"
server_key_version: "encryptionkey@0"
I20260811 12:34:06.012777 15903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:06.029544 15903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:06.035336 15903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:06.035712 15903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:06.037607 15903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:06.039981 15903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:06.040146 15903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:06.040314 15903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:06.040484 15903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:06.128820 15903 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:34057
I20260811 12:34:06.128979 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:34057 every 8 connection(s)
I20260811 12:34:06.130896 15903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:34:06.136552 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 15903
I20260811 12:34:06.137087 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 14894
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.124@KRBTEST.COM
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.125@KRBTEST.COM
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.66@KRBTEST.COM
I20260811 12:34:06.195926 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.135012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:49731 (local address 127.3.204.126:36811)
0811 12:34:06.151663 (+ 16651us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:06.151671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:06.161731 (+ 10060us) server_negotiation.cc:407] Connection header received
0811 12:34:06.167557 (+ 5826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:06.167564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:06.167655 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:06.167758 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:06.171647 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.173128 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.176804 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.177775 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.177889 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.185462 (+ 7573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:06.185473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:06.185479 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:06.185527 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:06.190075 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.191857 (+ 1782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.191863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.191866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.192294 (+ 428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.192988 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.192994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.192997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.193444 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:06.193586 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:06.195111 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0811 12:34:06.195386 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16468,"thread_start_us":16657,"threads_started":1}
I20260811 12:34:06.195956 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.135124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:55319 (local address 127.3.204.124:39971)
0811 12:34:06.135377 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:06.135385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:06.167126 (+ 31741us) server_negotiation.cc:407] Connection header received
0811 12:34:06.167700 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:06.167707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:06.167805 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:06.167884 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:06.172797 (+ 4913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.174029 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.176803 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.177828 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.177919 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.185427 (+ 7508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:06.185436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:06.185441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:06.185524 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:06.191372 (+ 5848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.192544 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.192550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.192554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.192801 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.193406 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.193411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.193415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.193890 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:06.194029 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:06.195063 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0811 12:34:06.195379 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 12:34:06.197821 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.144836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:39971 (local address 127.3.204.65:55319)
0811 12:34:06.166962 (+ 22126us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:06.166989 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:06.167186 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:06.168432 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:06.168442 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:06.169530 (+ 1088us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:06.172623 (+ 3093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.172636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.174249 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.174255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.176631 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.176641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.178363 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.178372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.178835 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.180011 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:06.180035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:06.185100 (+ 5065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:06.191564 (+ 6464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.191568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.191572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.192413 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.192942 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.192947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.192950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.193256 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.194115 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:06.194119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:06.194493 (+ 374us) client_negotiation.cc:769] Sending connection context
0811 12:34:06.196259 (+ 1766us) client_negotiation.cc:240] Negotiation successful
0811 12:34:06.196555 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20466,"mutex_wait_us":158,"spinlock_wait_cycles":7936,"thread_start_us":20129,"threads_started":1}
I20260811 12:34:06.197901 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.142763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.65:49731)
0811 12:34:06.161491 (+ 18728us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:06.161566 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:06.167187 (+ 5621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:06.168448 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:06.168453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:06.169547 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:06.171425 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.171467 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.173331 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.173336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.176602 (+ 3266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.176621 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.178493 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.178503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.178910 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.180151 (+ 1241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:06.180173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:06.185034 (+ 4861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:06.190235 (+ 5201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.190246 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.190267 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.191278 (+ 1011us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.192404 (+ 1126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.192416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.192420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.192832 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.193781 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:06.193789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:06.194545 (+ 756us) client_negotiation.cc:769] Sending connection context
0811 12:34:06.194889 (+ 344us) client_negotiation.cc:240] Negotiation successful
0811 12:34:06.195670 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13551,"mutex_wait_us":304,"spinlock_wait_cycles":24192,"thread_start_us":18231,"threads_started":1}
I20260811 12:34:06.198439 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.135033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:57009 (local address 127.3.204.125:32801)
0811 12:34:06.160805 (+ 25772us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:06.160813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:06.164154 (+ 3341us) server_negotiation.cc:407] Connection header received
0811 12:34:06.167673 (+ 3519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:06.167679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:06.167785 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:06.167887 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:06.171993 (+ 4106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.173553 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.176965 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.177905 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.178015 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.185897 (+ 7882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:06.185908 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:06.185913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:06.185962 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:06.191564 (+ 5602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.193383 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.193389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.193393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.193619 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.194197 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.194202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.194205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.194633 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:06.194788 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:06.197587 (+ 2799us) server_negotiation.cc:299] Negotiation successful
0811 12:34:06.197883 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25605,"thread_start_us":25850,"threads_started":1}
I20260811 12:34:06.198843 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.143912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.3.204.65:57009)
0811 12:34:06.162571 (+ 18659us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:06.162594 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:06.166606 (+ 4012us) spinlock_profiling.cc:243] Waited 873 us on lock 0x61200002c628. stack: 00007f3df647b66e 00007f3df647b430 00007f3df55025ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f3df64710fd 00007f3df6470d1e 00007f3e0937e3cd 00007f3e0937e36c 00007f3e0934ab3b 00007f3dfce80536 00007f3dfce7ed9e 00007f3dfcf06dca 00007f3dfcf04f68 00007f3dfcf3882f
0811 12:34:06.167505 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:06.169494 (+ 1989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:06.169499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:06.170703 (+ 1204us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:06.171790 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.171808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.174125 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.174137 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.176752 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.176771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.178291 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.178296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.178829 (+ 533us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.179720 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:06.179754 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:06.185534 (+ 5780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:06.191714 (+ 6180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.191719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.191723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.193227 (+ 1504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.193748 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.193752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.193755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.194043 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.196191 (+ 2148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:06.196196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:06.196386 (+ 190us) client_negotiation.cc:769] Sending connection context
0811 12:34:06.196614 (+ 228us) client_negotiation.cc:240] Negotiation successful
0811 12:34:06.196890 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18471,"mutex_wait_us":160,"spinlock_wait_cycles":2013056,"thread_start_us":12192,"threads_started":1}
I20260811 12:34:06.200024 16234 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36811
I20260811 12:34:06.200328 16235 heartbeater.cc:347] Connected to a master server at 127.3.204.125:32801
I20260811 12:34:06.200558 16235 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:06.200536 16234 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:06.201733 16233 heartbeater.cc:347] Connected to a master server at 127.3.204.124:39971
I20260811 12:34:06.201997 16233 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:06.202086 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.139112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.3.204.66:40033)
0811 12:34:06.162717 (+ 23605us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:06.162729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:06.163838 (+ 1109us) server_negotiation.cc:407] Connection header received
0811 12:34:06.164029 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:06.164041 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:06.164535 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:06.164726 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:06.168435 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.170613 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.175056 (+ 4443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.176403 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.176566 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.186458 (+ 9892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:06.186481 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:06.186503 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:06.186581 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:06.195258 (+ 8677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.197238 (+ 1980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.197249 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.197252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.197562 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.199217 (+ 1655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.199222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.199226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.200160 (+ 934us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:06.200552 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:06.201055 (+ 503us) server_negotiation.cc:299] Negotiation successful
0811 12:34:06.201421 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23344,"thread_start_us":22991,"threads_started":1}
I20260811 12:34:06.202874 16235 heartbeater.cc:511] Master 127.3.204.125:32801 requested a full tablet report, sending...
I20260811 12:34:06.203285 16233 heartbeater.cc:511] Master 127.3.204.124:39971 requested a full tablet report, sending...
I20260811 12:34:06.203598 15106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:06.204048 16234 heartbeater.cc:511] Master 127.3.204.126:36811 requested a full tablet report, sending...
I20260811 12:34:06.205719 14563 ts_manager.cc:194] Re-registered known tserver with Master: 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057)
I20260811 12:34:06.206322 15927 ts_manager.cc:194] Registered new tserver with Master: 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057)
I20260811 12:34:06.206843 14563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:49731
I20260811 12:34:06.207633 15569 ts_manager.cc:194] Registered new tserver with Master: 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057)
I20260811 12:34:06.294027 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 14894
I20260811 12:34:06.323915 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:40033
--local_ip_for_outbound_sockets=127.3.204.66
--tserver_master_addrs=127.3.204.126:36811
--webserver_port=34073
--webserver_interface=127.3.204.66
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab
--principal=oryx/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801,127.3.204.124:39971 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:34:06.514858 16014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: NON_VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: true } } }
I20260811 12:34:06.515241 16014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:06.516446 16014 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: attempting to promote NON_VOTER ccdba4578a1943138c8d21cd870a3978 to VOTER
I20260811 12:34:06.517127 16014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } }
I20260811 12:34:06.519371 15944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 LEARNER]: Refusing update from remote peer 917d930be6064324b7e8f465196a501e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260811 12:34:06.519433 15582 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Refusing update from remote peer 917d930be6064324b7e8f465196a501e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260811 12:34:06.519982 16260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260811 12:34:06.520442 16014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [LEADER]: Connected to new peer: Peer: permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260811 12:34:06.525839 15944 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ccdba4578a1943138c8d21cd870a3978 (127.3.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:06.524869 16260 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ccdba4578a1943138c8d21cd870a3978 (127.3.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:06.533119 16269 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:06.533838 16269 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:34:06.534757 15582 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ccdba4578a1943138c8d21cd870a3978 (127.3.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:06.535051 16014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:06.535385 16014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:06.535147 16261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:06.535561 16261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:06.535938 16260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:06.536190 16260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:06.538370 16270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "917d930be6064324b7e8f465196a501e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:06.538719 16270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 12:34:06.665355 16257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:06.665721 16257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:06.665880 16257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:06.673283 16257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:06.673425 16257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:06.673593 16257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:34:06.684197 16257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab
--principal=oryx/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:40033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=34073
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801,127.3.204.124:39971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:06.685734 16257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:06.687148 16257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:06.701258 16280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:06.701339 16281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:06.701303 16283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:06.702471 16257 server_base.cc:1034] running on GCE node
I20260811 12:34:06.703069 16257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:06.704056 16257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:06.705332 16257 hybrid_clock.cc:648] HybridClock initialized: now 1786451646705253 us; error 64 us; skew 500 ppm
I20260811 12:34:06.705709 16257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:06.709028 16257 init.cc:377] Logged in from keytab as oryx/127.3.204.66@KRBTEST.COM (short username oryx)
I20260811 12:34:06.710620 16257 webserver.cc:460] Webserver started at http://127.3.204.66:34073/ using document root <none> and password file <none>
I20260811 12:34:06.711349 16257 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:06.711486 16257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:06.717535 16257 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:34:06.720523 16295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:06.721956 16257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:34:06.722128 16257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "9000d312ec7242a4902ab42e3d31a271"
format_stamp: "Formatted at 2026-08-11 12:34:03 on dist-test-slave-lgr5"
server_key: "41f94e0e7a5177fd7caebf66b2d899b1"
server_key_iv: "94d314a70c5a404c8aafda3734c5560a"
server_key_version: "encryptionkey@0"
I20260811 12:34:06.723177 16257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:06.745136 16257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:06.749517 16257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:06.749799 16257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:06.750867 16257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:06.752337 16257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:06.752429 16257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:06.752543 16257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:06.752612 16257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:06.785223 16257 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:40033
I20260811 12:34:06.785398 16481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:40033 every 8 connection(s)
I20260811 12:34:06.786684 16257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:34:06.787776 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 16257
I20260811 12:34:06.788126 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 15126
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.67@KRBTEST.COM
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.125@KRBTEST.COM
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
Aug 11 12:34:06 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.124@KRBTEST.COM
I20260811 12:34:06.852790 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.789695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.3.204.67:42597)
0811 12:34:06.809478 (+ 19783us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:06.809489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:06.809531 (+ 42us) server_negotiation.cc:407] Connection header received
0811 12:34:06.809628 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:06.809638 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:06.809968 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:06.810161 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:06.824624 (+ 14463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.826953 (+ 2329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.829817 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.830998 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.831108 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.841007 (+ 9899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:06.841033 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:06.841055 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:06.841135 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:06.847017 (+ 5882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.848130 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.848142 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.848145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.848394 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.849023 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.849028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.849032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.849591 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:06.850029 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:06.851730 (+ 1701us) server_negotiation.cc:299] Negotiation successful
0811 12:34:06.852071 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19515,"thread_start_us":20125,"threads_started":1}
I20260811 12:34:06.854671 15342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:06.855726 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.805794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:39971 (local address 127.3.204.66:35433)
0811 12:34:06.827272 (+ 21478us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:06.827292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:06.827417 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:06.830858 (+ 3441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:06.830863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:06.832402 (+ 1539us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:06.833409 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.833426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.835740 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.835746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.838217 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.838234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.840264 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.840276 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.840643 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.841757 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:06.841781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:06.846132 (+ 4351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:06.851660 (+ 5528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.851665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.851669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.852506 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.853360 (+ 854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.853365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.853369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.853635 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.854377 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:06.854382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:06.854550 (+ 168us) client_negotiation.cc:769] Sending connection context
0811 12:34:06.854658 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 12:34:06.854984 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21304,"mutex_wait_us":119,"thread_start_us":21761,"threads_started":1}
I20260811 12:34:06.856484 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.791153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:38255 (local address 127.3.204.125:32801)
0811 12:34:06.802990 (+ 11837us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:06.802999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:06.816104 (+ 13105us) server_negotiation.cc:407] Connection header received
0811 12:34:06.816177 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:06.816182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:06.816345 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:06.816474 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:06.830109 (+ 13635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.831472 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.835061 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.836015 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.836171 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.844274 (+ 8103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:06.844283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:06.844289 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:06.844387 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:06.849283 (+ 4896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.850521 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.850526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.850530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.850821 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.851478 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.851483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.851485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.851907 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:06.852059 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:06.855615 (+ 3556us) server_negotiation.cc:299] Negotiation successful
0811 12:34:06.855926 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11673,"thread_start_us":11856,"threads_started":1}
I20260811 12:34:06.856735 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.790162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:60123 (local address 127.3.204.126:36811)
0811 12:34:06.808416 (+ 18254us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:06.808423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:06.816839 (+ 8416us) server_negotiation.cc:407] Connection header received
0811 12:34:06.817419 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:06.817425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:06.817554 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:06.817637 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:06.828171 (+ 10534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.829898 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.835063 (+ 5165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.836143 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.836304 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.844265 (+ 7961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:06.844272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:06.844278 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:06.844362 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:06.849955 (+ 5593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.851180 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.851186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.851189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.851467 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.852055 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.852060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.852063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.852483 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:06.852608 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:06.853330 (+ 722us) server_negotiation.cc:299] Negotiation successful
0811 12:34:06.853617 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18104,"thread_start_us":18263,"threads_started":1}
I20260811 12:34:06.855785 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.804847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:35433 (local address 127.3.204.124:39971)
0811 12:34:06.830495 (+ 25648us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:06.830501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:06.830521 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:34:06.830582 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:06.830587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:06.830675 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:06.830792 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:06.833620 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.835526 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.838420 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.839404 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.839535 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.846450 (+ 6915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:06.846459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:06.846465 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:06.846514 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:06.851498 (+ 4984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.852695 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.852701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.852705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.853205 (+ 500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:06.853798 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:06.853803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:06.853806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:06.854226 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:06.854413 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:06.854922 (+ 509us) server_negotiation.cc:299] Negotiation successful
0811 12:34:06.855205 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25473,"thread_start_us":25282,"threads_started":1}
I20260811 12:34:06.857673 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.790839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.3.204.66:38255)
0811 12:34:06.811628 (+ 20789us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:06.811708 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:06.815672 (+ 3964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:06.821160 (+ 5488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:06.821166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:06.821756 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:06.822836 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.822868 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.831667 (+ 8799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.831672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.834853 (+ 3181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.834870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.836716 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.836725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.837251 (+ 526us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.838402 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:06.838467 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:06.842805 (+ 4338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:06.849473 (+ 6668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.849482 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.849502 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.850358 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.850987 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.850992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.850995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.851321 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.852179 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:06.852188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:06.852758 (+ 570us) client_negotiation.cc:769] Sending connection context
0811 12:34:06.853129 (+ 371us) client_negotiation.cc:240] Negotiation successful
0811 12:34:06.853809 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16993,"mutex_wait_us":874,"spinlock_wait_cycles":40576,"thread_start_us":10509,"threads_started":1}
I20260811 12:34:06.858553 16483 heartbeater.cc:347] Connected to a master server at 127.3.204.124:39971
I20260811 12:34:06.858827 16485 heartbeater.cc:347] Connected to a master server at 127.3.204.125:32801
I20260811 12:34:06.859055 16485 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:06.860700 16485 heartbeater.cc:511] Master 127.3.204.125:32801 requested a full tablet report, sending...
I20260811 12:34:06.861025 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:06.790839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.66:60123)
0811 12:34:06.816734 (+ 25895us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:06.816754 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:06.817099 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:06.818231 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:06.818246 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:06.819761 (+ 1515us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:06.825738 (+ 5977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.825752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.831125 (+ 5373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.831131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.834882 (+ 3751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:06.834891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:06.836796 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:06.836804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:06.837253 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:06.838420 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:06.838497 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:06.843928 (+ 5431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:06.850158 (+ 6230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.850164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.850168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.851028 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.851621 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:06.851626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:06.851629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:06.851915 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:06.852716 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:06.852720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:06.852965 (+ 245us) client_negotiation.cc:769] Sending connection context
0811 12:34:06.860161 (+ 7196us) client_negotiation.cc:240] Negotiation successful
0811 12:34:06.860441 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12539,"mutex_wait_us":234,"spinlock_wait_cycles":96640,"thread_start_us":9274,"threads_started":1}
I20260811 12:34:06.862069 16484 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36811
I20260811 12:34:06.859061 16483 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:06.862283 16484 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:06.863082 15569 ts_manager.cc:194] Registered new tserver with Master: 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033)
I20260811 12:34:06.863534 16483 heartbeater.cc:511] Master 127.3.204.124:39971 requested a full tablet report, sending...
I20260811 12:34:06.863643 16484 heartbeater.cc:511] Master 127.3.204.126:36811 requested a full tablet report, sending...
I20260811 12:34:06.865105 15927 ts_manager.cc:194] Registered new tserver with Master: 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033)
I20260811 12:34:06.865120 14563 ts_manager.cc:194] Re-registered known tserver with Master: 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033)
I20260811 12:34:06.866305 14563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:60123
I20260811 12:34:06.953595 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 15126
I20260811 12:34:06.975139 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:42597
--local_ip_for_outbound_sockets=127.3.204.67
--tserver_master_addrs=127.3.204.126:36811
--webserver_port=40941
--webserver_interface=127.3.204.67
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab
--principal=oryx/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801,127.3.204.124:39971 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:34:07.210132 16234 heartbeater.cc:503] Master 127.3.204.126:36811 was elected leader, sending a full tablet report...
W20260811 12:34:07.319110 16508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:07.319409 16508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:07.319504 16508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:07.326699 16508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:07.326814 16508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:07.326960 16508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:34:07.337116 16508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab
--principal=oryx/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:42597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=40941
--tserver_master_addrs=127.3.204.126:36811,127.3.204.125:32801,127.3.204.124:39971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:07.338552 16508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:07.339895 16508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:07.353076 16526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:07.353122 16519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:07.353018 16518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:07.354240 16508 server_base.cc:1034] running on GCE node
I20260811 12:34:07.355034 16508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:07.356052 16508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:07.357307 16508 hybrid_clock.cc:648] HybridClock initialized: now 1786451647357235 us; error 68 us; skew 500 ppm
I20260811 12:34:07.357681 16508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:07 dist-test-slave-lgr5 krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451647, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:07.361074 16508 init.cc:377] Logged in from keytab as oryx/127.3.204.67@KRBTEST.COM (short username oryx)
I20260811 12:34:07.362649 16508 webserver.cc:460] Webserver started at http://127.3.204.67:40941/ using document root <none> and password file <none>
I20260811 12:34:07.363402 16508 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:07.363533 16508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:07.369261 16508 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:34:07.372062 16533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:07.373479 16508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:34:07.373615 16508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "cd5b7a24432e46baac22efd6d1d3ca30"
format_stamp: "Formatted at 2026-08-11 12:34:03 on dist-test-slave-lgr5"
server_key: "b18b55735b56e40cb4d9cf693da79745"
server_key_iv: "59fe8f0ae84a284ebea995390cab7b10"
server_key_version: "encryptionkey@0"
I20260811 12:34:07.374590 16508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:07.405243 16508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:07.409605 16508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:07.409879 16508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:07.410898 16508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:07.412398 16508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:07.412497 16508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:07.412585 16508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:07.412640 16508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:07.444293 16508 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:42597
I20260811 12:34:07.444438 16728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:42597 every 8 connection(s)
I20260811 12:34:07.445845 16508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:34:07.450600 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 16508
Aug 11 12:34:07 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451647, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
Aug 11 12:34:07 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451647, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.125@KRBTEST.COM
I20260811 12:34:07.505903 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.448861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:58587 (local address 127.3.204.126:36811)
0811 12:34:07.464579 (+ 15718us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.464585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.476435 (+ 11850us) server_negotiation.cc:407] Connection header received
0811 12:34:07.476621 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.476627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.476707 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.476800 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.480693 (+ 3893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.482008 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.485771 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.486803 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.486913 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.496915 (+ 10002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.496923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.496931 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.496971 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.501358 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.502428 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.502435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.502438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.502594 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.503255 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.503259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.503262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.503639 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.503772 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.505261 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.505500 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":15564,"thread_start_us":15718,"threads_started":1}
I20260811 12:34:07.506878 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.463853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.3.204.126:36811)
0811 12:34:07.469678 (+ 5825us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.469685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.469707 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:34:07.469788 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.469794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.469919 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.480838 (+ 10919us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.481123 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.483813 (+ 2690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.486746 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.488446 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.488594 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.496641 (+ 8047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.496649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.496653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.496700 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.501253 (+ 4553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.503164 (+ 1911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.503170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.503173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.503400 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.503927 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.503932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.503935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.504325 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.504571 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.505656 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.505880 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":5664,"thread_start_us":5436,"threads_started":1}
I20260811 12:34:07.507205 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.449578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.67:58587)
0811 12:34:07.476286 (+ 26708us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:07.476322 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:07.476488 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:07.477387 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:07.477402 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:07.478618 (+ 1216us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:07.480372 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.480411 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.482261 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.482266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.485572 (+ 3306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.485591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.487581 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.487594 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.488238 (+ 644us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.491250 (+ 3012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:07.491295 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:07.496607 (+ 5312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:07.501508 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:07.501524 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:07.501551 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:07.502275 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:07.502763 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:07.502767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:07.502771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:07.503133 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:07.503892 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:07.504194 (+ 302us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:07.504729 (+ 535us) client_negotiation.cc:769] Sending connection context
0811 12:34:07.505101 (+ 372us) client_negotiation.cc:240] Negotiation successful
0811 12:34:07.505677 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26500,"mutex_wait_us":64,"thread_start_us":40789,"threads_started":1}
I20260811 12:34:07.509035 16730 heartbeater.cc:347] Connected to a master server at 127.3.204.126:36811
I20260811 12:34:07.509358 16730 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:07.510281 16730 heartbeater.cc:511] Master 127.3.204.126:36811 requested a full tablet report, sending...
I20260811 12:34:07.513938 14563 ts_manager.cc:194] Re-registered known tserver with Master: cd5b7a24432e46baac22efd6d1d3ca30 (127.3.204.67:42597)
I20260811 12:34:07.515100 14563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:58587
Aug 11 12:34:07 dist-test-slave-lgr5 krb5kdc[14485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451647, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.124@KRBTEST.COM
I20260811 12:34:07.518230 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.450606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:50985 (local address 127.3.204.125:32801)
0811 12:34:07.474685 (+ 24079us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.474692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.474717 (+ 25us) server_negotiation.cc:407] Connection header received
0811 12:34:07.474788 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.474794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.474923 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.475068 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.492699 (+ 17631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.494847 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.497444 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.498112 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.498232 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.506403 (+ 8171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.506408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.506412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.506447 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.514380 (+ 7933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.515209 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.515215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.515219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.515392 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.515924 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.515929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.515932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.516602 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.516756 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.517442 (+ 686us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.517743 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23938,"thread_start_us":23784,"threads_started":1}
I20260811 12:34:07.519385 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.449578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.3.204.67:50985)
0811 12:34:07.470794 (+ 21216us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:07.470868 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:07.473691 (+ 2823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:07.487828 (+ 14137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:07.487833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:07.489118 (+ 1285us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:07.492505 (+ 3387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.492518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.495016 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.495023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.496840 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.496858 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.498579 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.498586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.498778 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.500812 (+ 2034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:07.500835 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:07.504762 (+ 3927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:07.514523 (+ 9761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:07.514528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:07.514533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:07.515025 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:07.515534 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:07.515539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:07.515542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:07.515728 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:07.516925 (+ 1197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:07.516930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:07.517155 (+ 225us) client_negotiation.cc:769] Sending connection context
0811 12:34:07.518497 (+ 1342us) client_negotiation.cc:240] Negotiation successful
0811 12:34:07.518913 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20869,"spinlock_wait_cycles":27904,"thread_start_us":26164,"threads_started":1}
I20260811 12:34:07.520313 16732 heartbeater.cc:347] Connected to a master server at 127.3.204.125:32801
I20260811 12:34:07.520465 16732 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:07.520872 16732 heartbeater.cc:511] Master 127.3.204.125:32801 requested a full tablet report, sending...
I20260811 12:34:07.522107 15569 ts_manager.cc:194] Registered new tserver with Master: cd5b7a24432e46baac22efd6d1d3ca30 (127.3.204.67:42597)
I20260811 12:34:07.514153 14562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 12:34:07.525698 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.465096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.3.204.125:32801)
0811 12:34:07.500109 (+ 35013us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.500115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.500134 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:34:07.500186 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.500191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.500267 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.500375 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.506588 (+ 6213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.509570 (+ 2982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.511543 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.512235 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.512358 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.518251 (+ 5893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.518259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.518264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.518302 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.522217 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.523181 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.523186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.523189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.523382 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.523817 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.523822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.523826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.524147 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.524391 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.525259 (+ 868us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.525436 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34833,"thread_start_us":36830,"threads_started":1}
I20260811 12:34:07.527181 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.450158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:49651 (local address 127.3.204.124:39971)
0811 12:34:07.468863 (+ 18705us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.468870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.491444 (+ 22574us) server_negotiation.cc:407] Connection header received
0811 12:34:07.491715 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.491721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.491823 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.491903 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.493777 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.495381 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.502289 (+ 6908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.503091 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.503187 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.519979 (+ 16792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.519988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.519992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.520037 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.523620 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.524547 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.524553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.524556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.524713 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.525131 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.525137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.525140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.525508 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.526096 (+ 588us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.526647 (+ 551us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.526883 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18547,"thread_start_us":18435,"threads_started":1}
I20260811 12:34:07.527436 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.451675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:39971 (local address 127.3.204.67:49651)
0811 12:34:07.491304 (+ 39629us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:07.491325 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:07.491565 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:07.492187 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:07.492192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:07.493214 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:07.493593 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.493606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.499099 (+ 5493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.499105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.500577 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.500591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.503498 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.503505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.503683 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.505016 (+ 1333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:07.505038 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:07.519719 (+ 14681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:07.523770 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:07.523775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:07.523779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:07.524357 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:07.524820 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:07.524825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:07.524829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:07.525001 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:07.526183 (+ 1182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:07.526187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:07.526350 (+ 163us) client_negotiation.cc:769] Sending connection context
0811 12:34:07.526504 (+ 154us) client_negotiation.cc:240] Negotiation successful
0811 12:34:07.526854 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39474,"thread_start_us":18642,"threads_started":1}
I20260811 12:34:07.528469 16729 heartbeater.cc:347] Connected to a master server at 127.3.204.124:39971
I20260811 12:34:07.528663 16729 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:07.528975 16729 heartbeater.cc:511] Master 127.3.204.124:39971 requested a full tablet report, sending...
W20260811 12:34:07.529210 14562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:34:07.530193 15927 ts_manager.cc:194] Registered new tserver with Master: cd5b7a24432e46baac22efd6d1d3ca30 (127.3.204.67:42597)
I20260811 12:34:07.538661 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.477307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.3.204.124:39971)
0811 12:34:07.498580 (+ 21273us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.498586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.502795 (+ 4209us) server_negotiation.cc:407] Connection header received
0811 12:34:07.502856 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.502861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.502961 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.503069 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.509594 (+ 6525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.510387 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.511744 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.512324 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.512455 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.530691 (+ 18236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.530724 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.530729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.530769 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.534211 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.535129 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.535135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.535139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.535332 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.535814 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.535818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.535821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.536276 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.537761 (+ 1485us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.537897 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.538175 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21072,"thread_start_us":21284,"threads_started":1}
I20260811 12:34:07.581899 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.565384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.3.204.66:40033)
0811 12:34:07.573149 (+ 7765us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.573156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.573179 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:34:07.573268 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.573273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.574194 (+ 921us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.574415 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:07.575574 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.577430 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.579878 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.580877 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.581016 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.581250 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.581451 (+ 201us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.581621 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7636,"thread_start_us":6778,"threads_started":1}
I20260811 12:34:07.581984 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.564812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:40033 (local address 127.0.0.1:55824)
0811 12:34:07.569068 (+ 4256us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:07.569088 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:07.569223 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:07.574743 (+ 5520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:07.574746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:07.574766 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:07.575167 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.575179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.577600 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.577606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.579709 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.579721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.581073 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.581080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.581259 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.581295 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:34:07.581386 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 12:34:07.581534 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4086,"thread_start_us":786,"threads_started":1}
I20260811 12:34:07.583037 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.562261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:42597 (local address 127.0.0.1:56240)
0811 12:34:07.567888 (+ 5627us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:07.567910 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:07.568048 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:07.573812 (+ 5764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:07.573818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:07.573850 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:07.574495 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.574513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.576546 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.576552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.579242 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.579259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.582274 (+ 3015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.582281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.582470 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.582505 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:34:07.582586 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 12:34:07.582715 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5488,"thread_start_us":5319,"threads_started":1}
I20260811 12:34:07.583310 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.562688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.3.204.67:42597)
0811 12:34:07.572723 (+ 10035us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.572734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.572776 (+ 42us) server_negotiation.cc:407] Connection header received
0811 12:34:07.572893 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.572903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.573581 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.573812 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:07.574681 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.576364 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.580259 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.581818 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.581941 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.582562 (+ 621us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.582635 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.582804 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9837,"thread_start_us":9626,"threads_started":1}
I20260811 12:34:07.585261 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.561811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:42990)
0811 12:34:07.565404 (+ 3593us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:07.565424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:07.565592 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:07.576990 (+ 11398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:07.576995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:07.577022 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:07.577616 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.577629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.580192 (+ 2563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.580197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.582230 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.582245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.584532 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.584539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.584721 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.584753 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 12:34:07.584819 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 12:34:07.584946 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3450,"thread_start_us":6879,"threads_started":1}
I20260811 12:34:07.586375 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.562050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.3.204.65:34057)
0811 12:34:07.574454 (+ 12404us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.574473 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.574502 (+ 29us) server_negotiation.cc:407] Connection header received
0811 12:34:07.574598 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.576519 (+ 1921us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.576784 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.578531 (+ 1747us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:07.578756 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.580047 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.582567 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.584236 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.585653 (+ 1417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.585859 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.585928 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.586101 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12230,"thread_start_us":13248,"threads_started":1}
I20260811 12:34:07.586812 16374 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd0adaaf8704e2ca396273776a0276b (DEFAULT_TABLE table=test-table [id=989ea900f46b463483e2ead135779862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:07.587096 16614 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd0adaaf8704e2ca396273776a0276b (DEFAULT_TABLE table=test-table [id=989ea900f46b463483e2ead135779862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:07.588267 16374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd0adaaf8704e2ca396273776a0276b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:07.588732 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd0adaaf8704e2ca396273776a0276b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:07.589964 16126 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd0adaaf8704e2ca396273776a0276b (DEFAULT_TABLE table=test-table [id=989ea900f46b463483e2ead135779862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:07.591367 16126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd0adaaf8704e2ca396273776a0276b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:07.600860 16789 tablet_bootstrap.cc:492] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: Bootstrap starting.
I20260811 12:34:07.604064 16789 tablet_bootstrap.cc:654] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:07.605294 16791 tablet_bootstrap.cc:492] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Bootstrap starting.
I20260811 12:34:07.606791 16792 tablet_bootstrap.cc:492] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Bootstrap starting.
I20260811 12:34:07.607466 16789 log.cc:824] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:07.609047 16791 tablet_bootstrap.cc:654] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:07.610067 16792 tablet_bootstrap.cc:654] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:07.610618 16791 log.cc:824] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:07.611020 16789 tablet_bootstrap.cc:492] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: No bootstrap required, opened a new log
I20260811 12:34:07.611109 16792 log.cc:824] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:07.611740 16789 ts_tablet_manager.cc:1397] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260811 12:34:07.614080 16791 tablet_bootstrap.cc:492] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: No bootstrap required, opened a new log
I20260811 12:34:07.614166 16792 tablet_bootstrap.cc:492] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: No bootstrap required, opened a new log
I20260811 12:34:07.614382 16791 ts_tablet_manager.cc:1397] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 12:34:07.614396 16792 ts_tablet_manager.cc:1397] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260811 12:34:07.620618 16789 raft_consensus.cc:348] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.621011 16789 raft_consensus.cc:374] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:07.621115 16789 raft_consensus.cc:729] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9000d312ec7242a4902ab42e3d31a271, State: Initialized, Role: FOLLOWER
I20260811 12:34:07.621711 16789 consensus_queue.cc:260] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.623466 16789 ts_tablet_manager.cc:1428] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260811 12:34:07.623736 16484 heartbeater.cc:503] Master 127.3.204.126:36811 was elected leader, sending a full tablet report...
I20260811 12:34:07.625408 16791 raft_consensus.cc:348] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.625880 16791 raft_consensus.cc:374] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:07.625988 16791 raft_consensus.cc:729] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5b7a24432e46baac22efd6d1d3ca30, State: Initialized, Role: FOLLOWER
I20260811 12:34:07.626680 16791 consensus_queue.cc:260] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.628319 16792 raft_consensus.cc:348] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.628844 16792 raft_consensus.cc:374] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:07.628966 16792 raft_consensus.cc:729] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db67396c57e416ca0503984086065d0, State: Initialized, Role: FOLLOWER
I20260811 12:34:07.629720 16792 consensus_queue.cc:260] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.630772 16791 ts_tablet_manager.cc:1428] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260811 12:34:07.631209 16730 heartbeater.cc:503] Master 127.3.204.126:36811 was elected leader, sending a full tablet report...
I20260811 12:34:07.637606 16792 ts_tablet_manager.cc:1428] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Time spent starting tablet: real 0.023s user 0.017s sys 0.003s
I20260811 12:34:07.651387 16798 raft_consensus.cc:482] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:34:07.651701 16798 raft_consensus.cc:504] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.653501 16798 leader_election.cc:288] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057), 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033)
W20260811 12:34:07.654656 16236 tablet.cc:2369] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:34:07.659354 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.654168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:58389 (local address 127.3.204.65:34057)
0811 12:34:07.654450 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.654455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.654476 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:34:07.654547 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.654552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.654630 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.654729 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:07.655122 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.655863 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.657566 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.658367 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.658529 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.658609 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.658971 (+ 362us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.659104 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 12:34:07.659390 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.653961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:47771 (local address 127.3.204.66:40033)
0811 12:34:07.654196 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.654202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.654224 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:34:07.654296 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.654300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.654358 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.654451 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:07.654958 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.655904 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.657596 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.658421 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.658581 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.658655 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.658994 (+ 339us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.659139 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 12:34:07.659623 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.653781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:40033 (local address 127.3.204.67:47771)
0811 12:34:07.654059 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:07.654073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:07.654148 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:07.654501 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:07.654506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:07.654534 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:07.654702 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.654713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.656062 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.656070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.657427 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.657441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.658595 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.658601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.658834 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.658862 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 12:34:07.658942 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 12:34:07.659096 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":105,"spinlock_wait_cycles":3328}
I20260811 12:34:07.659649 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.653781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.3.204.67:58389)
0811 12:34:07.653991 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:07.654012 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:07.654140 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:07.654758 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:07.654762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:07.654786 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:07.654960 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.654971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.656022 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.656026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.657397 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:07.657412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.658552 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.658558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:07.658811 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.658843 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 12:34:07.658935 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 12:34:07.659095 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":4}
I20260811 12:34:07.660552 16415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd0adaaf8704e2ca396273776a0276b" candidate_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9000d312ec7242a4902ab42e3d31a271" is_pre_election: true
I20260811 12:34:07.660552 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd0adaaf8704e2ca396273776a0276b" candidate_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db67396c57e416ca0503984086065d0" is_pre_election: true
I20260811 12:34:07.661018 16158 raft_consensus.cc:2454] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5b7a24432e46baac22efd6d1d3ca30 in term 0.
I20260811 12:34:07.661018 16415 raft_consensus.cc:2454] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5b7a24432e46baac22efd6d1d3ca30 in term 0.
I20260811 12:34:07.661648 16536 leader_election.cc:302] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9000d312ec7242a4902ab42e3d31a271, cd5b7a24432e46baac22efd6d1d3ca30; no voters:
I20260811 12:34:07.662030 16798 raft_consensus.cc:2781] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:34:07.662159 16798 raft_consensus.cc:482] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:34:07.662264 16798 raft_consensus.cc:3037] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:07.664772 16798 raft_consensus.cc:504] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.665356 16798 leader_election.cc:288] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [CANDIDATE]: Term 1 election: Requested vote from peers 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057), 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033)
I20260811 12:34:07.665887 16415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd0adaaf8704e2ca396273776a0276b" candidate_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9000d312ec7242a4902ab42e3d31a271"
I20260811 12:34:07.665906 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd0adaaf8704e2ca396273776a0276b" candidate_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db67396c57e416ca0503984086065d0"
I20260811 12:34:07.666127 16158 raft_consensus.cc:3037] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:07.666137 16415 raft_consensus.cc:3037] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:07.668589 16415 raft_consensus.cc:2454] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5b7a24432e46baac22efd6d1d3ca30 in term 1.
I20260811 12:34:07.668751 16158 raft_consensus.cc:2454] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5b7a24432e46baac22efd6d1d3ca30 in term 1.
I20260811 12:34:07.668998 16536 leader_election.cc:302] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9000d312ec7242a4902ab42e3d31a271, cd5b7a24432e46baac22efd6d1d3ca30; no voters:
I20260811 12:34:07.669320 16798 raft_consensus.cc:2781] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:07.669786 16798 raft_consensus.cc:686] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 LEADER]: Becoming Leader. State: Replica: cd5b7a24432e46baac22efd6d1d3ca30, State: Running, Role: LEADER
I20260811 12:34:07.670316 16798 consensus_queue.cc:237] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } }
I20260811 12:34:07.675541 14562 catalog_manager.cc:5495] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd5b7a24432e46baac22efd6d1d3ca30 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } health_report { overall_health: UNKNOWN } } }
W20260811 12:34:07.704254 16734 tablet.cc:2369] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:34:07.704645 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.699201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.3.204.67:42597)
0811 12:34:07.699448 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.699455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.699474 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:34:07.699583 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.699587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.699712 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.699854 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:07.700317 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.701092 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.702439 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.702851 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.702972 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.703465 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:07.704040 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:07.704239 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.704293 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.704384 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:34:07.712978 16158 raft_consensus.cc:1261] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 1 FOLLOWER]: Refusing update from remote peer cd5b7a24432e46baac22efd6d1d3ca30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:07.712978 16415 raft_consensus.cc:1261] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 1 FOLLOWER]: Refusing update from remote peer cd5b7a24432e46baac22efd6d1d3ca30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:07.713946 16803 consensus_queue.cc:1035] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:07.714321 16798 consensus_queue.cc:1035] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:07.728425 16808 mvcc.cc:205] Tried to move back new op lower bound from 7317305949022056448 to 7317305948861730816. Current Snapshot: MvccSnapshot[applied={T|T < 7317305949022056448}]
I20260811 12:34:07.746627 16811 mvcc.cc:205] Tried to move back new op lower bound from 7317305949022056448 to 7317305948861730816. Current Snapshot: MvccSnapshot[applied={T|T < 7317305949022056448}]
I20260811 12:34:07.747169 14563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:07.747435 14563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:07.755930 14563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 917d930be6064324b7e8f465196a501e: Sending DeleteTablet for 3 replicas of tablet 5cd0adaaf8704e2ca396273776a0276b
I20260811 12:34:07.757359 16614 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd0adaaf8704e2ca396273776a0276b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:07 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:56240
I20260811 12:34:07.757566 16374 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd0adaaf8704e2ca396273776a0276b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:07 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:55824
I20260811 12:34:07.761081 16126 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd0adaaf8704e2ca396273776a0276b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:07 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:42990
I20260811 12:34:07.768944 16823 tablet_replica.cc:330] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: stopping tablet replica
I20260811 12:34:07.769519 16823 raft_consensus.cc:2229] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:07.769912 16829 tablet_replica.cc:330] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: stopping tablet replica
I20260811 12:34:07.770789 16829 raft_consensus.cc:2229] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:07.771292 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 14503
I20260811 12:34:07.772388 16822 tablet_replica.cc:330] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: stopping tablet replica
I20260811 12:34:07.772711 14605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:07.772782 16822 raft_consensus.cc:2229] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:34:07.773481 16822 raft_consensus.cc:2258] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:07.773983 16823 raft_consensus.cc:2258] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:07.777009 16823 ts_tablet_manager.cc:1905] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:07.777107 16822 ts_tablet_manager.cc:1905] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:07.777693 16829 raft_consensus.cc:2258] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:07.783533 16829 ts_tablet_manager.cc:1905] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:07.785372 16822 ts_tablet_manager.cc:1918] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:07.785581 16822 log.cc:1196] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/5cd0adaaf8704e2ca396273776a0276b
I20260811 12:34:07.786145 16822 ts_tablet_manager.cc:1939] T 5cd0adaaf8704e2ca396273776a0276b P cd5b7a24432e46baac22efd6d1d3ca30: Deleting consensus metadata
I20260811 12:34:07.798372 16823 ts_tablet_manager.cc:1918] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:07.798557 16823 log.cc:1196] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/5cd0adaaf8704e2ca396273776a0276b
I20260811 12:34:07.808871 16823 ts_tablet_manager.cc:1939] T 5cd0adaaf8704e2ca396273776a0276b P 9000d312ec7242a4902ab42e3d31a271: Deleting consensus metadata
I20260811 12:34:07.815847 16829 ts_tablet_manager.cc:1918] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:07.816294 16829 log.cc:1196] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/5cd0adaaf8704e2ca396273776a0276b
I20260811 12:34:07.817031 16829 ts_tablet_manager.cc:1939] T 5cd0adaaf8704e2ca396273776a0276b P 9db67396c57e416ca0503984086065d0: Deleting consensus metadata
I20260811 12:34:07.831472 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.760486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.3.204.125:32801)
0811 12:34:07.760669 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.760676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.770204 (+ 9528us) server_negotiation.cc:407] Connection header received
0811 12:34:07.771386 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.771393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.771503 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.771595 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.804219 (+ 32624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.805361 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.807523 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.808278 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.808403 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.812143 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.812150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.812155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.812200 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.820838 (+ 8638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.821876 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.821881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.821886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.822045 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.822497 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.822502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.822506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.822883 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.823136 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.827341 (+ 4205us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.831116 (+ 3775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 12:34:07.844826 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.769894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.3.204.124:39971)
0811 12:34:07.770076 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.770083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.824463 (+ 54380us) server_negotiation.cc:407] Connection header received
0811 12:34:07.824769 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.824785 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.824878 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.824949 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.827530 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.828805 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.833116 (+ 4311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.833825 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.833927 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.837627 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.837634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.837639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.837688 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.841245 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.842193 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.842200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.842204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.842391 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.842846 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.842853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.842857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.843215 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.843462 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.843908 (+ 446us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.844198 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 12:34:07.920440 14540 catalog_manager.cc:4841] TS cd5b7a24432e46baac22efd6d1d3ca30 (127.3.204.67:42597): tablet 5cd0adaaf8704e2ca396273776a0276b (table test-table [id=989ea900f46b463483e2ead135779862]) successfully deleted
I20260811 12:34:07.919654 14544 catalog_manager.cc:4841] TS 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057): tablet 5cd0adaaf8704e2ca396273776a0276b (table test-table [id=989ea900f46b463483e2ead135779862]) successfully deleted
I20260811 12:34:07.920867 14538 catalog_manager.cc:4841] TS 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033): tablet 5cd0adaaf8704e2ca396273776a0276b (table test-table [id=989ea900f46b463483e2ead135779862]) successfully deleted
I20260811 12:34:07.921069 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 14503
I20260811 12:34:07.979044 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.953541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.3.204.124:39971)
0811 12:34:07.953750 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.953757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.962697 (+ 8940us) server_negotiation.cc:407] Connection header received
0811 12:34:07.962944 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.962951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.963062 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.963162 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.965636 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.966605 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.968107 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.968761 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.968899 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.972068 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.972076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.972114 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.972160 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.975601 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.976508 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.976513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.976517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.976664 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.977274 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.977280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.977283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.977658 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.977880 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.978465 (+ 585us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.978695 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:34:07.980968 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:07.954109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.3.204.125:32801)
0811 12:34:07.955210 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:07.955216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:07.958601 (+ 3385us) server_negotiation.cc:407] Connection header received
0811 12:34:07.959221 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:07.959229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:07.959336 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:07.959418 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:07.962968 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.964157 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.966389 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:07.967045 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:07.967195 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:07.970499 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:07.970504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:07.970508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:07.970543 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:07.975957 (+ 5414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.977606 (+ 1649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.977612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.977615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.977785 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:07.978494 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:07.978499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:07.978502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:07.978904 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:07.979289 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:07.980359 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0811 12:34:07.980599 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955}
I20260811 12:34:08.736328 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:08.735465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.66:58669)
0811 12:34:08.735955 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:08.736122 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
W20260811 12:34:08.736874 16484 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:36811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
I20260811 12:34:08.745154 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:08.744325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.67:51683)
0811 12:34:08.744815 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:08.744947 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
W20260811 12:34:08.745609 16730 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:36811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
I20260811 12:34:08.759121 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:08.758347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.65:53061)
0811 12:34:08.758814 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:08.758954 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
W20260811 12:34:08.759562 16234 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:36811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
I20260811 12:34:09.258781 16852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 917d930be6064324b7e8f465196a501e)
I20260811 12:34:09.259095 16852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } }
I20260811 12:34:09.260597 16852 leader_election.cc:288] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 917d930be6064324b7e8f465196a501e (127.3.204.126:36811), ccdba4578a1943138c8d21cd870a3978 (127.3.204.124:39971)
I20260811 12:34:09.261734 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.260753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39436)
0811 12:34:09.261290 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.261469 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":274,"threads_started":1}
W20260811 12:34:09.262032 15547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
I20260811 12:34:09.264021 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.263408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39450)
0811 12:34:09.263745 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.263828 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
W20260811 12:34:09.264371 15547 leader_election.cc:334] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 917d930be6064324b7e8f465196a501e (127.3.204.126:36811): Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
I20260811 12:34:09.265658 16863 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 917d930be6064324b7e8f465196a501e)
I20260811 12:34:09.265910 16863 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } }
I20260811 12:34:09.267319 16863 leader_election.cc:288] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 917d930be6064324b7e8f465196a501e (127.3.204.126:36811), 957dd776a08d4c5cab22af13198114e5 (127.3.204.125:32801)
I20260811 12:34:09.267632 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.260827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:39971 (local address 127.0.0.1:41452)
0811 12:34:09.261405 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.261423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:09.261587 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:09.262168 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:09.262173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:09.262211 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:09.262518 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.262534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.264064 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.264070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.265722 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.265737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.266871 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.266878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.267047 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.267073 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 12:34:09.267155 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 12:34:09.267299 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":415,"threads_started":1}
I20260811 12:34:09.267745 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.261104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.3.204.124:39971)
0811 12:34:09.261857 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.261864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.261886 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:34:09.261964 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.261968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.262030 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.262124 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:09.262775 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.263879 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.265876 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.266718 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.266837 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.266946 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.267195 (+ 249us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.267342 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":457,"threads_started":1}
I20260811 12:34:09.268464 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.267420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39458)
0811 12:34:09.267939 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.268146 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":533,"threads_started":1}
I20260811 12:34:09.268657 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "957dd776a08d4c5cab22af13198114e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ccdba4578a1943138c8d21cd870a3978" is_pre_election: true
I20260811 12:34:09.269011 15944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957dd776a08d4c5cab22af13198114e5 in term 1.
W20260811 12:34:09.269318 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
I20260811 12:34:09.270421 15548 leader_election.cc:302] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957dd776a08d4c5cab22af13198114e5, ccdba4578a1943138c8d21cd870a3978; no voters: 917d930be6064324b7e8f465196a501e
I20260811 12:34:09.270938 16852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 12:34:09.271070 16852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 917d930be6064324b7e8f465196a501e)
I20260811 12:34:09.271159 16852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 1 FOLLOWER]: Advancing to term 2
I20260811 12:34:09.271211 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.270699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39466)
0811 12:34:09.270865 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.271024 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 12:34:09.271377 15898 leader_election.cc:334] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 917d930be6064324b7e8f465196a501e (127.3.204.126:36811): Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
I20260811 12:34:09.273564 16852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } }
I20260811 12:34:09.274407 16852 leader_election.cc:288] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [CANDIDATE]: Term 2 election: Requested vote from peers 917d930be6064324b7e8f465196a501e (127.3.204.126:36811), ccdba4578a1943138c8d21cd870a3978 (127.3.204.124:39971)
I20260811 12:34:09.274940 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.274537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39482)
0811 12:34:09.274694 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.274790 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 12:34:09.274981 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "957dd776a08d4c5cab22af13198114e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ccdba4578a1943138c8d21cd870a3978"
I20260811 12:34:09.275251 15944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 1 FOLLOWER]: Advancing to term 2
I20260811 12:34:09.275835 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.268106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.3.204.125:32801)
0811 12:34:09.269303 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.269309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.269332 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:34:09.269390 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.269395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.269488 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.269590 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:09.270074 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.271210 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.273162 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.274468 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.275123 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.275248 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.275302 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.275426 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":913,"threads_started":1}
I20260811 12:34:09.276149 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.275687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39496)
0811 12:34:09.275882 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.275971 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260811 12:34:09.276165 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.267866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:32801 (local address 127.0.0.1:46608)
0811 12:34:09.268692 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.268711 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:09.268837 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:09.269612 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:09.269616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:09.269639 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:09.269883 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.269895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.271655 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.271659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.272980 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.272994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.274684 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.274691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.274886 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.274925 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 12:34:09.275066 (+ 141us) client_negotiation.cc:240] Negotiation successful
0811 12:34:09.275224 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":817,"threads_started":1}
W20260811 12:34:09.276311 15547 leader_election.cc:334] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 917d930be6064324b7e8f465196a501e (127.3.204.126:36811): Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
I20260811 12:34:09.277185 15582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccdba4578a1943138c8d21cd870a3978" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "957dd776a08d4c5cab22af13198114e5" is_pre_election: true
I20260811 12:34:09.277529 15582 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ccdba4578a1943138c8d21cd870a3978 in current term 2: Already voted for candidate 957dd776a08d4c5cab22af13198114e5 in this term.
I20260811 12:34:09.278072 15944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957dd776a08d4c5cab22af13198114e5 in term 2.
I20260811 12:34:09.278258 15895 leader_election.cc:302] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccdba4578a1943138c8d21cd870a3978; no voters: 917d930be6064324b7e8f465196a501e, 957dd776a08d4c5cab22af13198114e5
I20260811 12:34:09.278445 15548 leader_election.cc:302] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957dd776a08d4c5cab22af13198114e5, ccdba4578a1943138c8d21cd870a3978; no voters: 917d930be6064324b7e8f465196a501e
I20260811 12:34:09.278625 16863 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 12:34:09.278753 16852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 12:34:09.279449 16852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [term 2 LEADER]: Becoming Leader. State: Replica: 957dd776a08d4c5cab22af13198114e5, State: Running, Role: LEADER
I20260811 12:34:09.279838 16852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } }
I20260811 12:34:09.281728 16872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957dd776a08d4c5cab22af13198114e5. Latest consensus state: current_term: 2 leader_uuid: "957dd776a08d4c5cab22af13198114e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:09.282070 16872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:09.282684 16877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:34:09.286935 16877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:34:09.287844 16877 catalog_manager.cc:1259] Loaded cluster ID: 705d5cb53759433c9dfe5c8be7949bb0
I20260811 12:34:09.287921 16877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:34:09.289130 16877 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:34:09.290007 16877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Loaded TSK: 0
I20260811 12:34:09.290846 16877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:34:09.323516 15568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:34:09.330811 15568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:34:09.335101 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.334685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39518)
0811 12:34:09.334857 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.334942 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 12:34:09.335178 15944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [term 2 FOLLOWER]: Refusing update from remote peer 957dd776a08d4c5cab22af13198114e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260811 12:34:09.335902 16852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260811 12:34:09.336148 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.335717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:39526)
0811 12:34:09.335913 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.335987 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 12:34:09.336318 15547 consensus_peers.cc:487] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 -> Peer 917d930be6064324b7e8f465196a501e (127.3.204.126:36811): Couldn't send request to peer 917d930be6064324b7e8f465196a501e. Status: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 12:34:09.338693 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957dd776a08d4c5cab22af13198114e5. Latest consensus state: current_term: 2 leader_uuid: "957dd776a08d4c5cab22af13198114e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:09.338954 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:34:09.340435 16872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "957dd776a08d4c5cab22af13198114e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:09.340683 16872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5 [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:09.341244 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "957dd776a08d4c5cab22af13198114e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "917d930be6064324b7e8f465196a501e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36811 } } peers { permanent_uuid: "957dd776a08d4c5cab22af13198114e5" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 32801 } attrs { promote: false } } peers { permanent_uuid: "ccdba4578a1943138c8d21cd870a3978" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 39971 } attrs { promote: false } } }
I20260811 12:34:09.341544 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdba4578a1943138c8d21cd870a3978 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:34:09.356961 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.348956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:43004)
0811 12:34:09.349125 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.349149 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:09.349264 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:09.350334 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:09.350339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:09.350366 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:09.350607 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.350621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.352306 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.352313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.354309 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.354326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.356068 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.356094 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.356343 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.356387 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 12:34:09.356470 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 12:34:09.356625 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260811 12:34:09.358071 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.349075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.3.204.65:34057)
0811 12:34:09.349995 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.350001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.350020 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:34:09.350095 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.350100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.350171 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.350762 (+ 591us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:09.350925 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.352134 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.354563 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.355866 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.357455 (+ 1589us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.357603 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.357659 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.357807 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":699,"threads_started":1}
I20260811 12:34:09.358862 16126 tablet_service.cc:1467] Processing CreateTablet for tablet 1310609497a74c07a58e36a6f9916c45 (DEFAULT_TABLE table=test-table [id=26afa135f1a941bc820d31e583f86d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:09.359377 16126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1310609497a74c07a58e36a6f9916c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:09.364728 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.349439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:40033 (local address 127.0.0.1:55826)
0811 12:34:09.349609 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.349633 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:09.349741 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:09.359176 (+ 9435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:09.359181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:09.359207 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:09.359440 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.359451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.360729 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.360734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.362300 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.362313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.363834 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.363841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.364031 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.364067 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:34:09.364196 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 12:34:09.364362 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":49}
I20260811 12:34:09.365151 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.349758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:42597 (local address 127.0.0.1:56252)
0811 12:34:09.350263 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.350280 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:09.350426 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:09.358458 (+ 8032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:09.358463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:09.358494 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:09.358834 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.358847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.360315 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.360321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.362518 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:09.362534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.364228 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.364234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:09.364430 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.364465 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:34:09.364545 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 12:34:09.364682 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":34,"spinlock_wait_cycles":896,"thread_start_us":542,"threads_started":1}
I20260811 12:34:09.365559 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.349969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.3.204.66:40033)
0811 12:34:09.358905 (+ 8936us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.358910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.358925 (+ 15us) server_negotiation.cc:407] Connection header received
0811 12:34:09.358983 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.358987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.359041 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.359126 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:09.359611 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.360571 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.362460 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.363625 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.363761 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.363875 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.365104 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.365259 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8819,"thread_start_us":8720,"threads_started":1}
I20260811 12:34:09.366412 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.350161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.3.204.67:42597)
0811 12:34:09.358057 (+ 7896us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.358064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.358088 (+ 24us) server_negotiation.cc:407] Connection header received
0811 12:34:09.358169 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.358175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.358258 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.358424 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:09.359148 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.360150 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.362698 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.363970 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.365895 (+ 1925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.366017 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.366093 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.366214 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7737,"thread_start_us":7617,"threads_started":1}
I20260811 12:34:09.366377 16374 tablet_service.cc:1467] Processing CreateTablet for tablet 1310609497a74c07a58e36a6f9916c45 (DEFAULT_TABLE table=test-table [id=26afa135f1a941bc820d31e583f86d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:09.366870 16374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1310609497a74c07a58e36a6f9916c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:09.368659 16614 tablet_service.cc:1467] Processing CreateTablet for tablet 1310609497a74c07a58e36a6f9916c45 (DEFAULT_TABLE table=test-table [id=26afa135f1a941bc820d31e583f86d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:09.369052 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1310609497a74c07a58e36a6f9916c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:09.373993 16897 tablet_bootstrap.cc:492] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: Bootstrap starting.
I20260811 12:34:09.374176 16900 tablet_bootstrap.cc:492] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: Bootstrap starting.
I20260811 12:34:09.375609 16900 tablet_bootstrap.cc:654] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:09.375979 16897 tablet_bootstrap.cc:654] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:09.377048 16901 tablet_bootstrap.cc:492] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: Bootstrap starting.
I20260811 12:34:09.379026 16901 tablet_bootstrap.cc:654] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:09.381381 16901 tablet_bootstrap.cc:492] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: No bootstrap required, opened a new log
I20260811 12:34:09.381559 16901 ts_tablet_manager.cc:1397] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 12:34:09.381675 16897 tablet_bootstrap.cc:492] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: No bootstrap required, opened a new log
I20260811 12:34:09.381830 16897 ts_tablet_manager.cc:1397] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260811 12:34:09.382063 16900 tablet_bootstrap.cc:492] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: No bootstrap required, opened a new log
I20260811 12:34:09.382210 16900 ts_tablet_manager.cc:1397] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 12:34:09.382400 16901 raft_consensus.cc:348] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.382700 16901 raft_consensus.cc:374] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:09.382593 16897 raft_consensus.cc:348] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.382784 16901 raft_consensus.cc:729] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9000d312ec7242a4902ab42e3d31a271, State: Initialized, Role: FOLLOWER
I20260811 12:34:09.382817 16897 raft_consensus.cc:374] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:09.382896 16897 raft_consensus.cc:729] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db67396c57e416ca0503984086065d0, State: Initialized, Role: FOLLOWER
I20260811 12:34:09.382982 16900 raft_consensus.cc:348] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.383224 16900 raft_consensus.cc:374] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:09.383145 16897 consensus_queue.cc:260] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.383126 16901 consensus_queue.cc:260] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.383694 16900 raft_consensus.cc:729] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5b7a24432e46baac22efd6d1d3ca30, State: Initialized, Role: FOLLOWER
I20260811 12:34:09.384023 16900 consensus_queue.cc:260] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.385017 16897 ts_tablet_manager.cc:1428] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 12:34:09.386610 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.385628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.65:37531)
0811 12:34:09.386181 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.386342 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"threads_started":1}
I20260811 12:34:09.390358 16235 heartbeater.cc:503] Master 127.3.204.125:32801 was elected leader, sending a full tablet report...
I20260811 12:34:09.393226 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.392560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.65:56565)
0811 12:34:09.393017 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.393103 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
I20260811 12:34:09.384647 16901 ts_tablet_manager.cc:1428] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260811 12:34:09.393690 16485 heartbeater.cc:503] Master 127.3.204.125:32801 was elected leader, sending a full tablet report...
I20260811 12:34:09.393913 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.385606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.66:58015)
0811 12:34:09.387277 (+ 1671us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.387465 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1543,"thread_start_us":1395,"threads_started":1}
I20260811 12:34:09.396617 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.395418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.66:50953)
0811 12:34:09.396327 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.396445 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":796,"threads_started":1}
I20260811 12:34:09.398200 16900 ts_tablet_manager.cc:1428] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: Time spent starting tablet: real 0.016s user 0.005s sys 0.008s
I20260811 12:34:09.399703 16732 heartbeater.cc:503] Master 127.3.204.125:32801 was elected leader, sending a full tablet report...
I20260811 12:34:09.400151 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.397978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.67:48515)
0811 12:34:09.399703 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.399867 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1610,"thread_start_us":1469,"threads_started":1}
I20260811 12:34:09.401082 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.400709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.67:47369)
0811 12:34:09.400855 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.400948 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 12:34:09.500252 16913 raft_consensus.cc:482] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:34:09.500480 16913 raft_consensus.cc:504] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.501219 16913 leader_election.cc:288] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057), 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033)
I20260811 12:34:09.501876 16415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1310609497a74c07a58e36a6f9916c45" candidate_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9000d312ec7242a4902ab42e3d31a271" is_pre_election: true
I20260811 12:34:09.501935 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1310609497a74c07a58e36a6f9916c45" candidate_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db67396c57e416ca0503984086065d0" is_pre_election: true
I20260811 12:34:09.502121 16415 raft_consensus.cc:2454] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5b7a24432e46baac22efd6d1d3ca30 in term 0.
I20260811 12:34:09.502223 16158 raft_consensus.cc:2454] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5b7a24432e46baac22efd6d1d3ca30 in term 0.
I20260811 12:34:09.502722 16536 leader_election.cc:302] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9000d312ec7242a4902ab42e3d31a271, cd5b7a24432e46baac22efd6d1d3ca30; no voters:
I20260811 12:34:09.503003 16913 raft_consensus.cc:2781] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:34:09.503132 16913 raft_consensus.cc:482] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:34:09.503232 16913 raft_consensus.cc:3037] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:09.505360 16913 raft_consensus.cc:504] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.505995 16913 leader_election.cc:288] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [CANDIDATE]: Term 1 election: Requested vote from peers 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057), 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033)
I20260811 12:34:09.506588 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1310609497a74c07a58e36a6f9916c45" candidate_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db67396c57e416ca0503984086065d0"
I20260811 12:34:09.506659 16415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1310609497a74c07a58e36a6f9916c45" candidate_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9000d312ec7242a4902ab42e3d31a271"
I20260811 12:34:09.506814 16158 raft_consensus.cc:3037] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:09.506903 16415 raft_consensus.cc:3037] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:09.509042 16158 raft_consensus.cc:2454] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5b7a24432e46baac22efd6d1d3ca30 in term 1.
I20260811 12:34:09.509078 16415 raft_consensus.cc:2454] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5b7a24432e46baac22efd6d1d3ca30 in term 1.
I20260811 12:34:09.509488 16542 leader_election.cc:302] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9db67396c57e416ca0503984086065d0, cd5b7a24432e46baac22efd6d1d3ca30; no voters:
I20260811 12:34:09.509820 16913 raft_consensus.cc:2781] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:09.510233 16913 raft_consensus.cc:686] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 LEADER]: Becoming Leader. State: Replica: cd5b7a24432e46baac22efd6d1d3ca30, State: Running, Role: LEADER
I20260811 12:34:09.510540 16913 consensus_queue.cc:237] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } }
I20260811 12:34:09.511320 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.510884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.67:34573)
0811 12:34:09.511041 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.511154 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 12:34:09.513441 15569 catalog_manager.cc:5495] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd5b7a24432e46baac22efd6d1d3ca30 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5b7a24432e46baac22efd6d1d3ca30" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 42597 } health_report { overall_health: HEALTHY } } }
I20260811 12:34:09.556157 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.551191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.3.204.67:42597)
0811 12:34:09.551390 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.551395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.551413 (+ 18us) server_negotiation.cc:407] Connection header received
0811 12:34:09.551556 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.551560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.551624 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.551726 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:09.552151 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.552893 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.554234 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.554659 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.554753 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.555264 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:09.555407 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:09.555486 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.555727 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.555817 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 12:34:09.559746 16415 raft_consensus.cc:1261] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 1 FOLLOWER]: Refusing update from remote peer cd5b7a24432e46baac22efd6d1d3ca30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:09.559744 16158 raft_consensus.cc:1261] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 1 FOLLOWER]: Refusing update from remote peer cd5b7a24432e46baac22efd6d1d3ca30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:09.560469 16913 consensus_queue.cc:1035] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9000d312ec7242a4902ab42e3d31a271" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 40033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:09.560751 16920 consensus_queue.cc:1035] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db67396c57e416ca0503984086065d0" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:09.563879 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.563320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.65:35341)
0811 12:34:09.563569 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.563657 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260811 12:34:09.563911 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.562485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.66:58477)
0811 12:34:09.563358 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.563634 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702}
I20260811 12:34:09.571736 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.571193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.66:53233)
0811 12:34:09.571440 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.571554 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260811 12:34:09.572978 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.572518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.65:51189)
0811 12:34:09.572736 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.572827 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260811 12:34:09.573283 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.572698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.3.204.67:40823)
0811 12:34:09.573032 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.573124 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
I20260811 12:34:09.573696 16926 mvcc.cc:205] Tried to move back new op lower bound from 7317305956589518848 to 7317305956397830144. Current Snapshot: MvccSnapshot[applied={T|T < 7317305956589518848}]
I20260811 12:34:09.580204 15567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:09.580562 15567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:09.586520 15567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 957dd776a08d4c5cab22af13198114e5: Sending DeleteTablet for 3 replicas of tablet 1310609497a74c07a58e36a6f9916c45
I20260811 12:34:09.587600 16126 tablet_service.cc:1514] Processing DeleteTablet for tablet 1310609497a74c07a58e36a6f9916c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:09 UTC) from {username='oryx', principal='oryx/127.3.204.125@KRBTEST.COM'} at 127.0.0.1:43004
I20260811 12:34:09.587693 16374 tablet_service.cc:1514] Processing DeleteTablet for tablet 1310609497a74c07a58e36a6f9916c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:09 UTC) from {username='oryx', principal='oryx/127.3.204.125@KRBTEST.COM'} at 127.0.0.1:55826
I20260811 12:34:09.587800 16614 tablet_service.cc:1514] Processing DeleteTablet for tablet 1310609497a74c07a58e36a6f9916c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:09 UTC) from {username='oryx', principal='oryx/127.3.204.125@KRBTEST.COM'} at 127.0.0.1:56252
I20260811 12:34:09.588456 16944 tablet_replica.cc:330] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: stopping tablet replica
I20260811 12:34:09.588768 16944 raft_consensus.cc:2229] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:09.589185 16944 raft_consensus.cc:2258] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:09.589862 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 15903
I20260811 12:34:09.591073 16944 ts_tablet_manager.cc:1905] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:09.597872 16944 ts_tablet_manager.cc:1918] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:09.598134 16944 log.cc:1196] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/1310609497a74c07a58e36a6f9916c45
I20260811 12:34:09.598629 16944 ts_tablet_manager.cc:1939] T 1310609497a74c07a58e36a6f9916c45 P 9000d312ec7242a4902ab42e3d31a271: Deleting consensus metadata
I20260811 12:34:09.599913 15544 catalog_manager.cc:4841] TS 9000d312ec7242a4902ab42e3d31a271 (127.3.204.66:40033): tablet 1310609497a74c07a58e36a6f9916c45 (table test-table [id=26afa135f1a941bc820d31e583f86d48]) successfully deleted
I20260811 12:34:09.600659 16946 tablet_replica.cc:330] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: stopping tablet replica
I20260811 12:34:09.600876 16946 raft_consensus.cc:2229] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:34:09.601343 16946 raft_consensus.cc:2258] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:09.602828 16946 ts_tablet_manager.cc:1905] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:09.609364 16943 tablet_replica.cc:330] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: stopping tablet replica
I20260811 12:34:09.609598 16943 raft_consensus.cc:2229] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:09.609867 16946 ts_tablet_manager.cc:1918] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:09.609985 16943 raft_consensus.cc:2258] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:09.610023 16946 log.cc:1196] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/1310609497a74c07a58e36a6f9916c45
I20260811 12:34:09.610476 16946 ts_tablet_manager.cc:1939] T 1310609497a74c07a58e36a6f9916c45 P cd5b7a24432e46baac22efd6d1d3ca30: Deleting consensus metadata
I20260811 12:34:09.611219 16943 ts_tablet_manager.cc:1905] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:09.611717 15547 catalog_manager.cc:4841] TS cd5b7a24432e46baac22efd6d1d3ca30 (127.3.204.67:42597): tablet 1310609497a74c07a58e36a6f9916c45 (table test-table [id=26afa135f1a941bc820d31e583f86d48]) successfully deleted
I20260811 12:34:09.615434 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.591169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.3.204.65:34057)
0811 12:34:09.591407 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.591413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.598127 (+ 6714us) server_negotiation.cc:407] Connection header received
0811 12:34:09.599213 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.599221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.599322 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.599412 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:09.601489 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.602650 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.605538 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.606009 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.606135 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.608771 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:09.608787 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:09.608806 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:09.608851 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:09.612470 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:09.613215 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:09.613223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:09.613226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:09.613372 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:09.613758 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:09.613762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:09.613764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:09.614149 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:09.614482 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.614790 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.615116 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:34:09.616122 16228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:09.708756 16943 ts_tablet_manager.cc:1918] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:09.709018 16943 log.cc:1196] T 1310609497a74c07a58e36a6f9916c45 P 9db67396c57e416ca0503984086065d0: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/1310609497a74c07a58e36a6f9916c45
I20260811 12:34:09.709461 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 15903
I20260811 12:34:09.733071 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 16257
I20260811 12:34:09.733860 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.733351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:45516)
0811 12:34:09.733558 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.733676 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 12:34:09.734239 15548 catalog_manager.cc:4584] TS 9db67396c57e416ca0503984086065d0 (127.3.204.65:34057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1310609497a74c07a58e36a6f9916c45: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
I20260811 12:34:09.748070 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.733895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.3.204.66:40033)
0811 12:34:09.734120 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.734124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.734144 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:34:09.734260 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.734264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.734329 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.734423 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:09.735937 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.737088 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.738473 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.738940 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.739062 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.741932 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:09.741950 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:09.741968 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:09.742015 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:09.745104 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:09.745936 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:09.745945 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:09.745948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:09.746106 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:09.746546 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:09.746550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:09.746553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:09.746940 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:09.747184 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.747541 (+ 357us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.747739 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 12:34:09.748864 16476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:09.795136 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.794567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:45532)
0811 12:34:09.794808 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.794940 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 12:34:09.796296 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.795846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:45540)
0811 12:34:09.796007 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.796145 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 12:34:09.801476 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.801012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:56444)
0811 12:34:09.801180 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.801281 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 12:34:09.802552 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.802142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:36811 (local address 127.0.0.1:56448)
0811 12:34:09.802319 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.802408 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 12:34:09.855077 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 16257
I20260811 12:34:09.865087 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.864534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:45544)
0811 12:34:09.864742 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.864856 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 12:34:09.866299 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.865880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:45558)
0811 12:34:09.866060 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.866159 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 12:34:09.880498 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 16508
I20260811 12:34:09.895124 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.881289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.3.204.67:42597)
0811 12:34:09.881465 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:09.881470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:09.881490 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:34:09.881635 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:09.881639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:09.881705 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:09.881791 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:09.883178 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.883951 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.885385 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:09.885854 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:09.885969 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:09.888929 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:09.888947 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:09.888966 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:09.889010 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:09.892290 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:09.893031 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:09.893040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:09.893042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:09.893205 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:09.893586 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:09.893589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:09.893592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:09.894017 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:09.894246 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:09.894618 (+ 372us) server_negotiation.cc:299] Negotiation successful
0811 12:34:09.894815 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:34:09.896040 16721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:09.968575 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.968006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:45570)
0811 12:34:09.968266 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.968381 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260811 12:34:09.969676 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:09.969284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:45572)
0811 12:34:09.969438 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:09.969540 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 12:34:10.030335 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 16508
I20260811 12:34:10.060570 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 15357
I20260811 12:34:10.061713 15619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:10.189303 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:10.188469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:34057 (local address 127.0.0.1:45586)
0811 12:34:10.188759 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:10.188897 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260811 12:34:10.189870 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 15357
I20260811 12:34:10.218545 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 15508
I20260811 12:34:10.219718 15980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:10.337550 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 15508
2026-08-11T12:34:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10005 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:34:10 dist-test-slave-lgr5 krb5kdc[16960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:34:10 dist-test-slave-lgr5 krb5kdc[16960](info): set up 2 sockets
Aug 11 12:34:10 dist-test-slave-lgr5 krb5kdc[16960](info): commencing operation
krb5kdc: starting...
W20260811 12:34:11.610442 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:34:11 dist-test-slave-lgr5 krb5kdc[16960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:34:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.126.keytab.
Entry for principal oryx/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.126.keytab.
I20260811 12:34:11.868278 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:34375
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.126.keytab
--principal=oryx/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:12.209944 16981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:12.210260 16981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:12.210356 16981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:12.218119 16981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:12.218253 16981 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 12:34:12.218295 16981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:12.218380 16981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:12.218456 16981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:12.218533 16981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:12.229303 16981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:34375
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.126.keytab
--principal=oryx/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:12.230705 16981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:12.232272 16981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:12.245227 16990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:12.245230 16995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:12.245958 16981 server_base.cc:1034] running on GCE node
W20260811 12:34:12.246141 16992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:12.247212 16981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:12.248893 16981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:12.250128 16981 hybrid_clock.cc:648] HybridClock initialized: now 1786451652250051 us; error 54 us; skew 500 ppm
I20260811 12:34:12.250449 16981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:12 dist-test-slave-lgr5 krb5kdc[16960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451652, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:12.253681 16981 init.cc:377] Logged in from keytab as oryx/127.3.204.126@KRBTEST.COM (short username oryx)
I20260811 12:34:12.255259 16981 webserver.cc:460] Webserver started at http://127.3.204.126:33519/ using document root <none> and password file <none>
I20260811 12:34:12.255973 16981 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:12.256059 16981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:12.256397 16981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:12.259397 16981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "cff04494a32b4672a5be284cb5d06e7a"
format_stamp: "Formatted at 2026-08-11 12:34:12 on dist-test-slave-lgr5"
server_key: "8a455b4262406a4105850ef55a98fd89"
server_key_iv: "72f76d1d306b3d0820c6b04ecdb2e75f"
server_key_version: "encryptionkey@0"
I20260811 12:34:12.260234 16981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "cff04494a32b4672a5be284cb5d06e7a"
format_stamp: "Formatted at 2026-08-11 12:34:12 on dist-test-slave-lgr5"
server_key: "8a455b4262406a4105850ef55a98fd89"
server_key_iv: "72f76d1d306b3d0820c6b04ecdb2e75f"
server_key_version: "encryptionkey@0"
I20260811 12:34:12.265746 16981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 12:34:12.269632 17008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:12.271272 16981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:34:12.271462 16981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "cff04494a32b4672a5be284cb5d06e7a"
format_stamp: "Formatted at 2026-08-11 12:34:12 on dist-test-slave-lgr5"
server_key: "8a455b4262406a4105850ef55a98fd89"
server_key_iv: "72f76d1d306b3d0820c6b04ecdb2e75f"
server_key_version: "encryptionkey@0"
I20260811 12:34:12.271692 16981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:12.302901 16981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:12.307502 16981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:12.307790 16981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:12.327999 16981 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:34375
I20260811 12:34:12.328020 17111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:34375 every 8 connection(s)
I20260811 12:34:12.329417 16981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:34:12.333482 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 16981
I20260811 12:34:12.333669 17112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:12.333705 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:34:12.334503 3889 external_mini_cluster.cc:1428] Setting key a06f7168486a406b2faf24df70b2d7a3
I20260811 12:34:12.347847 17112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: Bootstrap starting.
Aug 11 12:34:12 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:12.351722 17112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:12.352998 17112 log.cc:824] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:12.356189 17112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: No bootstrap required, opened a new log
I20260811 12:34:12.359478 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:12.336754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.3.204.126:34375)
0811 12:34:12.337585 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:12.337597 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:12.337636 (+ 39us) server_negotiation.cc:407] Connection header received
0811 12:34:12.340034 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:12.340061 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:12.340550 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:12.340918 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:12.342751 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.344275 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.346096 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.346797 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.346937 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:12.351648 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:12.351657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:12.351673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:12.351719 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:12.355317 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:12.356048 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:12.356056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:12.356063 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:12.356293 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:12.356768 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:12.356772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:12.356774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:12.357308 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:12.357545 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:12.358039 (+ 494us) server_negotiation.cc:299] Negotiation successful
0811 12:34:12.358416 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":410,"threads_started":1}
I20260811 12:34:12.364897 17112 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff04494a32b4672a5be284cb5d06e7a" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34375 } }
I20260811 12:34:12.365294 17112 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:12.365392 17112 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff04494a32b4672a5be284cb5d06e7a, State: Initialized, Role: FOLLOWER
I20260811 12:34:12.365955 17112 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff04494a32b4672a5be284cb5d06e7a" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34375 } }
I20260811 12:34:12.366192 17112 raft_consensus.cc:388] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:34:12.366321 17112 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:34:12.366470 17112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:12.368860 17112 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff04494a32b4672a5be284cb5d06e7a" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34375 } }
I20260811 12:34:12.369422 17112 leader_election.cc:302] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff04494a32b4672a5be284cb5d06e7a; no voters:
I20260811 12:34:12.370010 17112 leader_election.cc:288] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:34:12.370191 17119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:12.370898 17119 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [term 1 LEADER]: Becoming Leader. State: Replica: cff04494a32b4672a5be284cb5d06e7a, State: Running, Role: LEADER
I20260811 12:34:12.371423 17119 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff04494a32b4672a5be284cb5d06e7a" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34375 } }
I20260811 12:34:12.372112 17112 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:12.374390 17121 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff04494a32b4672a5be284cb5d06e7a. Latest consensus state: current_term: 1 leader_uuid: "cff04494a32b4672a5be284cb5d06e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff04494a32b4672a5be284cb5d06e7a" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34375 } } }
I20260811 12:34:12.374825 17121 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:12.374785 17120 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cff04494a32b4672a5be284cb5d06e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff04494a32b4672a5be284cb5d06e7a" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34375 } } }
I20260811 12:34:12.375110 17120 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:12.376669 17131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:34:12.381975 17131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:34:12.391059 17131 catalog_manager.cc:1347] Generated new cluster ID: 04ed2cfa147d40ddaaed089946d79fd5
I20260811 12:34:12.391180 17131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:34:12.406206 17131 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:34:12.407555 17131 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:34:12.420753 17131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: Generated new TSK 0
I20260811 12:34:12.421635 17131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab.
Entry for principal oryx/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab.
I20260811 12:34:12.481932 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34375
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab
--principal=oryx/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:12.819365 17160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:12.819679 17160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:12.819802 17160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:12.827056 17160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:12.827163 17160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:12.827275 17160 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:34:12.827373 17160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:34:12.837342 17160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab
--principal=oryx/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:12.838627 17160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:12.839980 17160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:12.852864 17168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:12.852934 17169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:12.853072 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:12.853534 17160 server_base.cc:1034] running on GCE node
I20260811 12:34:12.854169 17160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:12.855227 17160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:12.856443 17160 hybrid_clock.cc:648] HybridClock initialized: now 1786451652856386 us; error 39 us; skew 500 ppm
I20260811 12:34:12.856740 17160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:12 dist-test-slave-lgr5 krb5kdc[16960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451652, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:12.859819 17160 init.cc:377] Logged in from keytab as oryx/127.3.204.65@KRBTEST.COM (short username oryx)
I20260811 12:34:12.861351 17160 webserver.cc:460] Webserver started at http://127.3.204.65:42687/ using document root <none> and password file <none>
I20260811 12:34:12.862030 17160 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:12.862118 17160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:12.862394 17160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:12.865396 17160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "72d6e43777814bc39a42b13d33e58c5a"
format_stamp: "Formatted at 2026-08-11 12:34:12 on dist-test-slave-lgr5"
server_key: "0d7ab1472e9c14d74b90a80be3ab6d92"
server_key_iv: "102b730ab0e9a5d6e7e660c38e52599b"
server_key_version: "encryptionkey@0"
I20260811 12:34:12.866173 17160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "72d6e43777814bc39a42b13d33e58c5a"
format_stamp: "Formatted at 2026-08-11 12:34:12 on dist-test-slave-lgr5"
server_key: "0d7ab1472e9c14d74b90a80be3ab6d92"
server_key_iv: "102b730ab0e9a5d6e7e660c38e52599b"
server_key_version: "encryptionkey@0"
I20260811 12:34:12.871217 17160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260811 12:34:12.874897 17186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:12.876434 17160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:34:12.876605 17160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "72d6e43777814bc39a42b13d33e58c5a"
format_stamp: "Formatted at 2026-08-11 12:34:12 on dist-test-slave-lgr5"
server_key: "0d7ab1472e9c14d74b90a80be3ab6d92"
server_key_iv: "102b730ab0e9a5d6e7e660c38e52599b"
server_key_version: "encryptionkey@0"
I20260811 12:34:12.876821 17160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:12.907271 17160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:12.911684 17160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:12.911964 17160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:12.914718 17160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:12.914814 17160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:12.914973 17160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:12.915045 17160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:34:12 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451652, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:12.942858 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:12.915558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:45391 (local address 127.3.204.126:34375)
0811 12:34:12.924436 (+ 8878us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:12.924441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:12.924462 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:34:12.924524 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:12.924528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:12.924595 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:12.924709 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:12.926983 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.928176 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.930724 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.931351 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.931505 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:12.936870 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:12.936877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:12.936881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:12.936926 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:12.939370 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:12.940147 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:12.940151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:12.940154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:12.940290 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:12.940776 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:12.940781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:12.940784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:12.941163 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:12.941304 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:12.942252 (+ 948us) server_negotiation.cc:299] Negotiation successful
0811 12:34:12.942495 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8757,"thread_start_us":8653,"threads_started":1}
I20260811 12:34:12.944399 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:12.916195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34375 (local address 127.3.204.65:45391)
0811 12:34:12.917067 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:12.917129 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:12.919036 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:12.925027 (+ 5991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:12.925039 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:12.925834 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:12.926681 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:12.926714 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.928367 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.928373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:12.930514 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:12.930543 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.931667 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.931677 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:12.931923 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:12.932859 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:12.932898 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:12.936598 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:12.939520 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:12.939529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:12.939548 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:12.940005 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:12.940411 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:12.940417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:12.940421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:12.940658 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:12.941320 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:12.941332 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:12.941760 (+ 428us) client_negotiation.cc:769] Sending connection context
0811 12:34:12.942139 (+ 379us) client_negotiation.cc:240] Negotiation successful
0811 12:34:12.942765 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":365,"threads_started":1}
I20260811 12:34:12.955938 17160 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:37507
I20260811 12:34:12.955962 17396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:37507 every 8 connection(s)
I20260811 12:34:12.957710 17160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:34:12.960132 17398 heartbeater.cc:347] Connected to a master server at 127.3.204.126:34375
I20260811 12:34:12.960391 17398 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:12.961036 17398 heartbeater.cc:511] Master 127.3.204.126:34375 requested a full tablet report, sending...
I20260811 12:34:12.963331 17035 ts_manager.cc:194] Registered new tserver with Master: 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507)
I20260811 12:34:12.965278 17035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:45391
I20260811 12:34:12.966183 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:12.951331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:59443 (local address 127.3.204.126:34375)
0811 12:34:12.951554 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:12.951562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:12.952764 (+ 1202us) server_negotiation.cc:407] Connection header received
0811 12:34:12.953097 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:12.953103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:12.953193 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:12.953303 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:12.954616 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.955663 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.957598 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.958104 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.958247 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:12.960432 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:12.960440 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:12.960446 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:12.960495 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:12.963331 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:12.964168 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:12.964174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:12.964177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:12.964365 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:12.964753 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:12.964756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:12.964759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:12.965053 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:12.965161 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:12.965667 (+ 506us) server_negotiation.cc:299] Negotiation successful
0811 12:34:12.965875 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:34:12.966250 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:12.951333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34375 (local address 127.3.204.65:59443)
0811 12:34:12.952638 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:12.952663 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:12.952903 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:12.953363 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:12.953367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:12.954074 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:12.954467 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:12.954481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.955848 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.955857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:12.957427 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:12.957443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:12.958364 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:12.958373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:12.958605 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:12.959261 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:12.959292 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:12.960206 (+ 914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:12.963472 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:12.963478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:12.963482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:12.963997 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:12.964478 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:12.964483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:12.964487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:12.964646 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:12.965162 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:12.965168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:12.965371 (+ 203us) client_negotiation.cc:769] Sending connection context
0811 12:34:12.965518 (+ 147us) client_negotiation.cc:240] Negotiation successful
0811 12:34:12.965829 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"thread_start_us":956,"threads_started":1}
I20260811 12:34:12.967865 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 17160
I20260811 12:34:12.968061 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:34:12.968878 3889 external_mini_cluster.cc:1428] Setting key 27509b6d04b63efd61ba8221c98147b8
WARNING: no policy specified for oryx/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab.
Entry for principal oryx/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab.
I20260811 12:34:13.024776 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34375
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab
--principal=oryx/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:13.362936 17408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:13.363265 17408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:13.363404 17408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:13.370922 17408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:13.371081 17408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:13.371203 17408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:34:13.371317 17408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:34:13.381981 17408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab
--principal=oryx/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:13.383417 17408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:13.384900 17408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:13.400390 17417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:13.400309 17419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:13.400334 17421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:13.401291 17408 server_base.cc:1034] running on GCE node
I20260811 12:34:13.402004 17408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:13.403229 17408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:13.404452 17408 hybrid_clock.cc:648] HybridClock initialized: now 1786451653404390 us; error 44 us; skew 500 ppm
I20260811 12:34:13.404775 17408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:13 dist-test-slave-lgr5 krb5kdc[16960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451653, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:13.408038 17408 init.cc:377] Logged in from keytab as oryx/127.3.204.66@KRBTEST.COM (short username oryx)
I20260811 12:34:13.409677 17408 webserver.cc:460] Webserver started at http://127.3.204.66:39025/ using document root <none> and password file <none>
I20260811 12:34:13.410398 17408 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:13.410502 17408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:13.410830 17408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:13.413875 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b"
format_stamp: "Formatted at 2026-08-11 12:34:13 on dist-test-slave-lgr5"
server_key: "7a9dcbd181dc9cde0e7de27b4301c0a5"
server_key_iv: "4102e3140c27a2584f3bf80b53d35e58"
server_key_version: "encryptionkey@0"
I20260811 12:34:13.414665 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b"
format_stamp: "Formatted at 2026-08-11 12:34:13 on dist-test-slave-lgr5"
server_key: "7a9dcbd181dc9cde0e7de27b4301c0a5"
server_key_iv: "4102e3140c27a2584f3bf80b53d35e58"
server_key_version: "encryptionkey@0"
I20260811 12:34:13.419983 17408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260811 12:34:13.423662 17433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:13.425050 17408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 12:34:13.425212 17408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b"
format_stamp: "Formatted at 2026-08-11 12:34:13 on dist-test-slave-lgr5"
server_key: "7a9dcbd181dc9cde0e7de27b4301c0a5"
server_key_iv: "4102e3140c27a2584f3bf80b53d35e58"
server_key_version: "encryptionkey@0"
I20260811 12:34:13.425431 17408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:13.437304 17408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:13.441217 17408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:13.441465 17408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:13.444177 17408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:13.444283 17408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:13.444439 17408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:13.444520 17408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:34:13 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451653, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:13.467983 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:13.444672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:33857 (local address 127.3.204.126:34375)
0811 12:34:13.444872 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:13.444879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:13.446377 (+ 1498us) server_negotiation.cc:407] Connection header received
0811 12:34:13.448958 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:13.448965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:13.449058 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:13.449151 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:13.451458 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:13.452683 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:13.454411 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:13.455138 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:13.455286 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:13.460171 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:13.460179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:13.460185 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:13.460236 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:13.463598 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:13.464595 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:13.464601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:13.464605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:13.464747 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:13.465218 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:13.465222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:13.465224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:13.465529 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:13.465681 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:13.466490 (+ 809us) server_negotiation.cc:299] Negotiation successful
0811 12:34:13.466752 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 12:34:13.469089 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:13.445340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34375 (local address 127.3.204.66:33857)
0811 12:34:13.446210 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:13.446252 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:13.448642 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:13.449513 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:13.449525 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:13.450302 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:13.451195 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:13.451221 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:13.452882 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:13.452887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:13.454250 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:13.454264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:13.455430 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:13.455435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:13.455656 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:13.456569 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:13.456595 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:13.459775 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:13.463754 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:13.463768 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:13.463792 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:13.464464 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:13.464867 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:13.464871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:13.464874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:13.465093 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:13.465661 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:13.465669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:13.466079 (+ 410us) client_negotiation.cc:769] Sending connection context
0811 12:34:13.466341 (+ 262us) client_negotiation.cc:240] Negotiation successful
0811 12:34:13.466910 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":393,"threads_started":1}
I20260811 12:34:13.489746 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:13.475993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:58773 (local address 127.3.204.126:34375)
0811 12:34:13.476202 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:13.476209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:13.476909 (+ 700us) server_negotiation.cc:407] Connection header received
0811 12:34:13.477120 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:13.477126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:13.477197 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:13.477277 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:13.478303 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:13.479029 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:13.480435 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:13.480995 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:13.481113 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:13.483126 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:13.483132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:13.483135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:13.483171 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:13.486584 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:13.487357 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:13.487363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:13.487367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:13.487543 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:13.487892 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:13.487897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:13.487900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:13.488432 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:13.488597 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:13.489330 (+ 733us) server_negotiation.cc:299] Negotiation successful
0811 12:34:13.489516 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 12:34:13.490104 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:13.476060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34375 (local address 127.3.204.66:58773)
0811 12:34:13.476808 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:13.476825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:13.476983 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:13.477318 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:13.477322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:13.477844 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:13.478152 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:13.478162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:13.479161 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:13.479166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:13.480291 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:13.480308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:13.481192 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:13.481198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:13.481382 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:13.482089 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:13.482106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:13.482866 (+ 760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:13.486742 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:13.486750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:13.486754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:13.487228 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:13.487653 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:13.487657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:13.487659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:13.487789 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:13.488670 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:13.488675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:13.488976 (+ 301us) client_negotiation.cc:769] Sending connection context
0811 12:34:13.489180 (+ 204us) client_negotiation.cc:240] Negotiation successful
0811 12:34:13.489601 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":419,"threads_started":1}
I20260811 12:34:13.500660 17408 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:38391
I20260811 12:34:13.500679 17631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:38391 every 8 connection(s)
I20260811 12:34:13.501909 17408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:34:13.504010 17634 heartbeater.cc:347] Connected to a master server at 127.3.204.126:34375
I20260811 12:34:13.504276 17634 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:13.504956 17634 heartbeater.cc:511] Master 127.3.204.126:34375 requested a full tablet report, sending...
I20260811 12:34:13.506335 17035 ts_manager.cc:194] Registered new tserver with Master: 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391)
I20260811 12:34:13.507175 17035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:33857
I20260811 12:34:13.511036 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 17408
I20260811 12:34:13.511178 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:34:13.511591 3889 external_mini_cluster.cc:1428] Setting key 50b7e1fbabf6b6f42457c851692bea8f
WARNING: no policy specified for oryx/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab.
Entry for principal oryx/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab.
I20260811 12:34:13.568854 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:34375
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab
--principal=oryx/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:13.901743 17640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:13.902060 17640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:13.902191 17640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:13.909655 17640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:13.909782 17640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:13.909919 17640 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 12:34:13.910043 17640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:34:13.920667 17640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab
--principal=oryx/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.204.126:34375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:13.922027 17640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:13.923408 17640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:13.936972 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:13.937014 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:13.936932 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:13.943212 17640 server_base.cc:1034] running on GCE node
I20260811 12:34:13.943908 17640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:13.944991 17640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:13.946254 17640 hybrid_clock.cc:648] HybridClock initialized: now 1786451653946178 us; error 43 us; skew 500 ppm
I20260811 12:34:13.946610 17640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:13 dist-test-slave-lgr5 krb5kdc[16960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451653, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:13.949849 17640 init.cc:377] Logged in from keytab as oryx/127.3.204.67@KRBTEST.COM (short username oryx)
I20260811 12:34:13.951334 17640 webserver.cc:460] Webserver started at http://127.3.204.67:42709/ using document root <none> and password file <none>
I20260811 12:34:13.952149 17640 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:13.952260 17640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:13.952596 17640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:13.955883 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "2b89b55c1c404c7db3e7776c5cc7f266"
format_stamp: "Formatted at 2026-08-11 12:34:13 on dist-test-slave-lgr5"
server_key: "f180f046ed06169fd2d810416c0faf70"
server_key_iv: "443023720c47186cd443ef8c6cb56443"
server_key_version: "encryptionkey@0"
I20260811 12:34:13.956856 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "2b89b55c1c404c7db3e7776c5cc7f266"
format_stamp: "Formatted at 2026-08-11 12:34:13 on dist-test-slave-lgr5"
server_key: "f180f046ed06169fd2d810416c0faf70"
server_key_iv: "443023720c47186cd443ef8c6cb56443"
server_key_version: "encryptionkey@0"
I20260811 12:34:13.962436 17640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260811 12:34:13.966065 17660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:13.967542 17640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:34:13.967724 17640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "2b89b55c1c404c7db3e7776c5cc7f266"
format_stamp: "Formatted at 2026-08-11 12:34:13 on dist-test-slave-lgr5"
server_key: "f180f046ed06169fd2d810416c0faf70"
server_key_iv: "443023720c47186cd443ef8c6cb56443"
server_key_version: "encryptionkey@0"
I20260811 12:34:13.967960 17640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:13.969177 17398 heartbeater.cc:503] Master 127.3.204.126:34375 was elected leader, sending a full tablet report...
I20260811 12:34:13.985999 17640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:13.990093 17640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:13.990371 17640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:13.993119 17640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:13.993206 17640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:13.993345 17640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:13.993463 17640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 11 12:34:14 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451653, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:14.016738 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:13.993635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:57085 (local address 127.3.204.126:34375)
0811 12:34:13.994697 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:13.994704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:13.995471 (+ 767us) server_negotiation.cc:407] Connection header received
0811 12:34:13.997563 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:13.997569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:13.997668 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:13.997822 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:13.999943 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.000956 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.003416 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.004054 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.004232 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:14.009237 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:14.009246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:14.009251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:14.009301 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:14.013100 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:14.013865 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:14.013871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:14.013876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:14.014066 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:14.014632 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:14.014637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:14.014640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:14.015031 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:14.015193 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:14.016001 (+ 808us) server_negotiation.cc:299] Negotiation successful
0811 12:34:14.016296 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":821,"threads_started":1}
I20260811 12:34:14.018203 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:13.994363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34375 (local address 127.3.204.67:57085)
0811 12:34:13.995300 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:13.995350 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:13.997277 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:13.998101 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:13.998112 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:13.998872 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:13.999733 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:13.999768 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.001132 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.001137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:14.003208 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:14.003228 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.004360 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.004367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:14.004590 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:14.005408 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:14.005443 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:14.008892 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:14.013267 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:14.013276 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:14.013290 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:14.013746 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:14.014235 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:14.014260 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:14.014263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:14.014528 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:14.015217 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:14.015223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:14.015634 (+ 411us) client_negotiation.cc:769] Sending connection context
0811 12:34:14.015884 (+ 250us) client_negotiation.cc:240] Negotiation successful
0811 12:34:14.016516 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":383,"threads_started":1}
I20260811 12:34:14.039340 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:14.024227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34375 (local address 127.3.204.67:44715)
0811 12:34:14.025250 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:14.025278 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:14.025447 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:14.025784 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:14.025789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:14.026378 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:14.026813 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:14.026832 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.028360 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.028365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:14.029454 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:14.029467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.030589 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.030594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:14.030708 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:14.031452 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:14.031485 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:14.032595 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:14.036553 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:14.036559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:14.036566 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:14.037170 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:14.037675 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:14.037680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:14.037684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:14.037821 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:14.038422 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:14.038425 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:14.038591 (+ 166us) client_negotiation.cc:769] Sending connection context
0811 12:34:14.038702 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 12:34:14.038971 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":684,"threads_started":1}
I20260811 12:34:14.039492 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:14.024241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44715 (local address 127.3.204.126:34375)
0811 12:34:14.024428 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:14.024434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:14.025370 (+ 936us) server_negotiation.cc:407] Connection header received
0811 12:34:14.025558 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:14.025562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:14.025629 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:14.025722 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:14.026999 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.028229 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.029593 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.030288 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.030461 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:14.032822 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:14.032828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:14.032832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:14.032874 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:14.036417 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:14.037326 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:14.037332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:14.037335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:14.037531 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:14.037923 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:14.037928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:14.037932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:14.038308 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:14.038499 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:14.038830 (+ 331us) server_negotiation.cc:299] Negotiation successful
0811 12:34:14.039091 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:34:14.045030 17640 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:39335
I20260811 12:34:14.045074 17866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:39335 every 8 connection(s)
I20260811 12:34:14.046218 17640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:34:14.048308 17871 heartbeater.cc:347] Connected to a master server at 127.3.204.126:34375
I20260811 12:34:14.048516 17871 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:14.049099 17871 heartbeater.cc:511] Master 127.3.204.126:34375 requested a full tablet report, sending...
I20260811 12:34:14.050606 17035 ts_manager.cc:194] Registered new tserver with Master: 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335)
I20260811 12:34:14.051414 17035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:57085
I20260811 12:34:14.053965 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 17640
I20260811 12:34:14.054100 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:34:14.054489 3889 external_mini_cluster.cc:1428] Setting key dbaada6cc72c3cb5f8f23a6b4625855a
I20260811 12:34:14.057497 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 12:34:14 dist-test-slave-lgr5 krb5kdc[16960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451654, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 12:34:14 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:14 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:14.509720 17634 heartbeater.cc:503] Master 127.3.204.126:34375 was elected leader, sending a full tablet report...
I20260811 12:34:15.053922 17871 heartbeater.cc:503] Master 127.3.204.126:34375 was elected leader, sending a full tablet report...
W20260811 12:34:15.129482 17883 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.020s user 0.005s sys 0.012s
W20260811 12:34:15.130878 17677 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:14.103861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.3.204.126:34375)
0811 12:34:14.104048 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:14.104053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:14.104349 (+ 296us) server_negotiation.cc:407] Connection header received
0811 12:34:14.104536 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:14.104541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:14.104613 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:14.104708 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:14.106080 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.106852 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.108166 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:14.108633 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:14.108735 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.130573 (+1021838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57160: BlockingRecv error: recv got EOF from 127.0.0.1:57160 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 11 12:34:15 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451654, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:15.152837 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.135427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.3.204.126:34375)
0811 12:34:15.135636 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.135643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.135904 (+ 261us) server_negotiation.cc:407] Connection header received
0811 12:34:15.136116 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.136123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.136199 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.136286 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:15.137519 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.138332 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.139605 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.140043 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.140182 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.147506 (+ 7324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:15.147512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:15.147515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:15.147560 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:15.150131 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:15.150842 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:15.150846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:15.150849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:15.150997 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:15.151371 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:15.151375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:15.151377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:15.151672 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:15.151908 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.152284 (+ 376us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.152538 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 12:34:15.157191 17035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:34:15.164674 17035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:34:15.189069 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.179079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.3.204.65:37507)
0811 12:34:15.179685 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.179693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.181331 (+ 1638us) server_negotiation.cc:407] Connection header received
0811 12:34:15.181436 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.181444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.181686 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.181868 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:15.182659 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.184313 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.186786 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.187963 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.188113 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.188335 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.188595 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.188767 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":288,"threads_started":1}
I20260811 12:34:15.189146 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.178644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:37507 (local address 127.0.0.1:42010)
0811 12:34:15.181069 (+ 2425us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:15.181086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:15.181205 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:15.182128 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:15.182133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:15.182157 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:15.182485 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.182498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.184500 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.184505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.186601 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.186616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.188185 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.188191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.188368 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.188413 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 12:34:15.188535 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 12:34:15.188743 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2277,"mutex_wait_us":1,"thread_start_us":2118,"threads_started":1}
I20260811 12:34:15.189823 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.178373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:38391 (local address 127.0.0.1:41816)
0811 12:34:15.179546 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:15.179604 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:15.179866 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:15.181046 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:15.181059 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:15.181090 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:15.181715 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.181732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.183588 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.183594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.186138 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.186154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.188755 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.188762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.188931 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.188962 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 12:34:15.189058 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 12:34:15.189197 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":786,"threads_started":1}
I20260811 12:34:15.190148 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.181128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:39335 (local address 127.0.0.1:51070)
0811 12:34:15.181728 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:15.181747 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:15.181864 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:15.183730 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:15.183734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:15.183763 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:15.184204 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.184216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.185520 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.185528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.187382 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.187398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.189427 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.189435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.189611 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.189645 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 12:34:15.189729 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 12:34:15.189858 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":542,"threads_started":1}
I20260811 12:34:15.190277 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.181520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.3.204.67:39335)
0811 12:34:15.183015 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.183024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.183050 (+ 26us) server_negotiation.cc:407] Connection header received
0811 12:34:15.183133 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.183140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.183329 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.183513 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:15.184409 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.185376 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.187575 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.188969 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.189093 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.189274 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.189815 (+ 541us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.189986 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":1203,"threads_started":1}
I20260811 12:34:15.192432 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.178555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.3.204.66:38391)
0811 12:34:15.179115 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.179124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.179716 (+ 592us) server_negotiation.cc:407] Connection header received
0811 12:34:15.180142 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.180151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.180481 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.180735 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:15.181918 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.183424 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.186367 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.188258 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.188403 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.188606 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.190358 (+ 1752us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.192218 (+ 1860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":260,"threads_started":1}
I20260811 12:34:15.193508 17276 tablet_service.cc:1467] Processing CreateTablet for tablet c97b7d5cc83f4ac396061d6b7320f7ae (DEFAULT_TABLE table=test-table [id=6d058d3e92d2476bb9470532e31da4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:15.194288 17749 tablet_service.cc:1467] Processing CreateTablet for tablet c97b7d5cc83f4ac396061d6b7320f7ae (DEFAULT_TABLE table=test-table [id=6d058d3e92d2476bb9470532e31da4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:15.195186 17276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97b7d5cc83f4ac396061d6b7320f7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.195152 17512 tablet_service.cc:1467] Processing CreateTablet for tablet c97b7d5cc83f4ac396061d6b7320f7ae (DEFAULT_TABLE table=test-table [id=6d058d3e92d2476bb9470532e31da4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:15.195650 17749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97b7d5cc83f4ac396061d6b7320f7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.196391 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97b7d5cc83f4ac396061d6b7320f7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.207026 17919 tablet_bootstrap.cc:492] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Bootstrap starting.
I20260811 12:34:15.210731 17919 tablet_bootstrap.cc:654] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.211470 17921 tablet_bootstrap.cc:492] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Bootstrap starting.
I20260811 12:34:15.212271 17919 log.cc:824] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:15.215073 17921 tablet_bootstrap.cc:654] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.215626 17919 tablet_bootstrap.cc:492] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: No bootstrap required, opened a new log
I20260811 12:34:15.215915 17919 ts_tablet_manager.cc:1397] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 12:34:15.216148 17920 tablet_bootstrap.cc:492] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Bootstrap starting.
I20260811 12:34:15.216362 17921 log.cc:824] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:15.219480 17920 tablet_bootstrap.cc:654] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.219594 17921 tablet_bootstrap.cc:492] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: No bootstrap required, opened a new log
I20260811 12:34:15.219877 17921 ts_tablet_manager.cc:1397] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260811 12:34:15.220721 17920 log.cc:824] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:15.223321 17920 tablet_bootstrap.cc:492] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: No bootstrap required, opened a new log
I20260811 12:34:15.223569 17920 ts_tablet_manager.cc:1397] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260811 12:34:15.226106 17919 raft_consensus.cc:348] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.226568 17919 raft_consensus.cc:374] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.226713 17919 raft_consensus.cc:729] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b3c5bbbbb246e4ac7e8946ca26ee0b, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.227392 17919 consensus_queue.cc:260] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.229290 17919 ts_tablet_manager.cc:1428] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260811 12:34:15.230240 17921 raft_consensus.cc:348] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.230760 17921 raft_consensus.cc:374] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.230865 17921 raft_consensus.cc:729] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b89b55c1c404c7db3e7776c5cc7f266, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.231518 17921 consensus_queue.cc:260] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.233531 17921 ts_tablet_manager.cc:1428] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260811 12:34:15.233430 17920 raft_consensus.cc:348] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.233875 17920 raft_consensus.cc:374] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.233944 17920 raft_consensus.cc:729] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d6e43777814bc39a42b13d33e58c5a, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.234556 17920 consensus_queue.cc:260] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.236681 17920 ts_tablet_manager.cc:1428] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260811 12:34:15.254769 17637 tablet.cc:2369] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:34:15.298676 17873 tablet.cc:2369] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:34:15.459098 17935 raft_consensus.cc:482] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:34:15.459474 17935 raft_consensus.cc:504] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
W20260811 12:34:15.461254 17399 tablet.cc:2369] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:34:15.461376 17935 leader_election.cc:288] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391), 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335)
I20260811 12:34:15.468163 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.462369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:58673 (local address 127.3.204.66:38391)
0811 12:34:15.462554 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.462560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.462734 (+ 174us) server_negotiation.cc:407] Connection header received
0811 12:34:15.462952 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.462957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.463018 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.463094 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:15.463690 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.464482 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.466362 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.467214 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.467371 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.467462 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.467750 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.467879 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 12:34:15.468459 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.462063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:38333 (local address 127.3.204.67:39335)
0811 12:34:15.462292 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.462298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.462995 (+ 697us) server_negotiation.cc:407] Connection header received
0811 12:34:15.463167 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.463171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.463230 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.463302 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:15.463891 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.464663 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.466463 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.467260 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.467410 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.467539 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.468024 (+ 485us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.468211 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 12:34:15.468545 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.461608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:38391 (local address 127.3.204.65:58673)
0811 12:34:15.462605 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:15.462623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:15.462803 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:15.463128 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:15.463132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:15.463158 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:15.463547 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.463557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.464608 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.464613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.466198 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.466212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.467442 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.467450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.467592 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.467618 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 12:34:15.467700 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 12:34:15.467870 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":1,"thread_start_us":665,"threads_started":1}
I20260811 12:34:15.468987 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.461739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:39335 (local address 127.3.204.65:38333)
0811 12:34:15.462885 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:15.462899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:15.463039 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:15.463345 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:15.463349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:15.463374 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:15.463741 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.463751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.464803 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.464807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.466306 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.466319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.467489 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.467496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.467771 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.467858 (+ 87us) client_negotiation.cc:769] Sending connection context
0811 12:34:15.467964 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 12:34:15.468430 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":42,"thread_start_us":847,"threads_started":1}
I20260811 12:34:15.469650 17545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97b7d5cc83f4ac396061d6b7320f7ae" candidate_uuid: "72d6e43777814bc39a42b13d33e58c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" is_pre_election: true
I20260811 12:34:15.469918 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97b7d5cc83f4ac396061d6b7320f7ae" candidate_uuid: "72d6e43777814bc39a42b13d33e58c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" is_pre_election: true
I20260811 12:34:15.470161 17545 raft_consensus.cc:2454] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d6e43777814bc39a42b13d33e58c5a in term 0.
I20260811 12:34:15.470415 17787 raft_consensus.cc:2454] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d6e43777814bc39a42b13d33e58c5a in term 0.
I20260811 12:34:15.470849 17194 leader_election.cc:302] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b3c5bbbbb246e4ac7e8946ca26ee0b, 72d6e43777814bc39a42b13d33e58c5a; no voters:
I20260811 12:34:15.471258 17935 raft_consensus.cc:2781] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:34:15.471419 17935 raft_consensus.cc:482] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:34:15.471549 17935 raft_consensus.cc:3037] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:15.474422 17935 raft_consensus.cc:504] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.475314 17935 leader_election.cc:288] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [CANDIDATE]: Term 1 election: Requested vote from peers 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391), 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335)
I20260811 12:34:15.475628 17545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97b7d5cc83f4ac396061d6b7320f7ae" candidate_uuid: "72d6e43777814bc39a42b13d33e58c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b"
I20260811 12:34:15.475781 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97b7d5cc83f4ac396061d6b7320f7ae" candidate_uuid: "72d6e43777814bc39a42b13d33e58c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b89b55c1c404c7db3e7776c5cc7f266"
I20260811 12:34:15.475860 17545 raft_consensus.cc:3037] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:15.475999 17787 raft_consensus.cc:3037] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:15.478389 17787 raft_consensus.cc:2454] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d6e43777814bc39a42b13d33e58c5a in term 1.
I20260811 12:34:15.478387 17545 raft_consensus.cc:2454] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d6e43777814bc39a42b13d33e58c5a in term 1.
I20260811 12:34:15.478811 17192 leader_election.cc:302] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b89b55c1c404c7db3e7776c5cc7f266, 72d6e43777814bc39a42b13d33e58c5a; no voters:
I20260811 12:34:15.479173 17935 raft_consensus.cc:2781] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:15.479650 17935 raft_consensus.cc:686] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 1 LEADER]: Becoming Leader. State: Replica: 72d6e43777814bc39a42b13d33e58c5a, State: Running, Role: LEADER
I20260811 12:34:15.480453 17935 consensus_queue.cc:237] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.485184 17035 catalog_manager.cc:5495] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65). New cstate: current_term: 1 leader_uuid: "72d6e43777814bc39a42b13d33e58c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } health_report { overall_health: UNKNOWN } } }
I20260811 12:34:15.558497 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.553412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.3.204.65:37507)
0811 12:34:15.553611 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.553616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.553656 (+ 40us) server_negotiation.cc:407] Connection header received
0811 12:34:15.553866 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.553869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.553948 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.554082 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:15.554519 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.555254 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.556625 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.557019 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.557111 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.557519 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:15.558053 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:15.558160 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.558214 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.558279 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:34:15.567005 17545 raft_consensus.cc:1261] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Refusing update from remote peer 72d6e43777814bc39a42b13d33e58c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:15.567005 17787 raft_consensus.cc:1261] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Refusing update from remote peer 72d6e43777814bc39a42b13d33e58c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:15.568248 17940 consensus_queue.cc:1035] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:15.568687 17935 consensus_queue.cc:1035] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:15.594738 17034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:15.595055 17034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:15.598884 17034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: Sending DeleteTablet for 3 replicas of tablet c97b7d5cc83f4ac396061d6b7320f7ae
I20260811 12:34:15.600225 17512 tablet_service.cc:1514] Processing DeleteTablet for tablet c97b7d5cc83f4ac396061d6b7320f7ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:15 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:41816
I20260811 12:34:15.600389 17276 tablet_service.cc:1514] Processing DeleteTablet for tablet c97b7d5cc83f4ac396061d6b7320f7ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:15 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:42010
I20260811 12:34:15.600425 17749 tablet_service.cc:1514] Processing DeleteTablet for tablet c97b7d5cc83f4ac396061d6b7320f7ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:15 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:51070
I20260811 12:34:15.602679 17034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:34:15.603434 17034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:34:15.614238 17511 tablet_service.cc:1467] Processing CreateTablet for tablet 40c29cf7743441bca552eef8151ef882 (DEFAULT_TABLE table=test-table [id=e01baa9f954449129da56a01eff4a366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:15.614555 17748 tablet_service.cc:1467] Processing CreateTablet for tablet 40c29cf7743441bca552eef8151ef882 (DEFAULT_TABLE table=test-table [id=e01baa9f954449129da56a01eff4a366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:15.614918 17511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c29cf7743441bca552eef8151ef882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.615168 17748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c29cf7743441bca552eef8151ef882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.615542 17971 tablet_replica.cc:330] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: stopping tablet replica
I20260811 12:34:15.616329 17971 raft_consensus.cc:2229] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:15.616616 17275 tablet_service.cc:1467] Processing CreateTablet for tablet 40c29cf7743441bca552eef8151ef882 (DEFAULT_TABLE table=test-table [id=e01baa9f954449129da56a01eff4a366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:15.617192 17275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c29cf7743441bca552eef8151ef882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.617774 17971 raft_consensus.cc:2258] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:15.621479 17971 ts_tablet_manager.cc:1905] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:15.622123 17919 tablet_bootstrap.cc:492] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Bootstrap starting.
I20260811 12:34:15.623556 17970 tablet_replica.cc:330] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: stopping tablet replica
I20260811 12:34:15.623813 17919 tablet_bootstrap.cc:654] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.624042 17970 raft_consensus.cc:2229] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:15.625029 17921 tablet_bootstrap.cc:492] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: Bootstrap starting.
I20260811 12:34:15.625854 17970 raft_consensus.cc:2258] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:15.626693 17921 tablet_bootstrap.cc:654] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.628930 17971 ts_tablet_manager.cc:1918] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:15.629096 17971 log.cc:1196] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/c97b7d5cc83f4ac396061d6b7320f7ae
I20260811 12:34:15.629665 17971 ts_tablet_manager.cc:1939] T c97b7d5cc83f4ac396061d6b7320f7ae P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting consensus metadata
I20260811 12:34:15.630537 17970 ts_tablet_manager.cc:1905] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:15.631171 17920 tablet_bootstrap.cc:492] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: Bootstrap starting.
I20260811 12:34:15.631554 17011 catalog_manager.cc:4841] TS 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335): tablet c97b7d5cc83f4ac396061d6b7320f7ae (table test-table [id=6d058d3e92d2476bb9470532e31da4ad]) successfully deleted
I20260811 12:34:15.632954 17920 tablet_bootstrap.cc:654] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.635076 17969 tablet_replica.cc:330] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: stopping tablet replica
I20260811 12:34:15.635553 17969 raft_consensus.cc:2229] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:34:15.636015 17920 tablet_bootstrap.cc:492] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: No bootstrap required, opened a new log
I20260811 12:34:15.636262 17920 ts_tablet_manager.cc:1397] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260811 12:34:15.637148 17920 raft_consensus.cc:348] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.637472 17920 raft_consensus.cc:374] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.637624 17920 raft_consensus.cc:729] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d6e43777814bc39a42b13d33e58c5a, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.637668 17921 tablet_bootstrap.cc:492] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: No bootstrap required, opened a new log
I20260811 12:34:15.637831 17921 ts_tablet_manager.cc:1397] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260811 12:34:15.638012 17920 consensus_queue.cc:260] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.638669 17920 ts_tablet_manager.cc:1428] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 12:34:15.636294 17969 raft_consensus.cc:2258] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:15.639137 17921 raft_consensus.cc:348] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.639362 17970 ts_tablet_manager.cc:1918] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:15.639391 17921 raft_consensus.cc:374] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.639483 17921 raft_consensus.cc:729] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b89b55c1c404c7db3e7776c5cc7f266, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.639524 17970 log.cc:1196] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/c97b7d5cc83f4ac396061d6b7320f7ae
I20260811 12:34:15.639770 17921 consensus_queue.cc:260] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.640060 17970 ts_tablet_manager.cc:1939] T c97b7d5cc83f4ac396061d6b7320f7ae P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting consensus metadata
I20260811 12:34:15.640403 17921 ts_tablet_manager.cc:1428] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 12:34:15.642249 17012 catalog_manager.cc:4841] TS 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391): tablet c97b7d5cc83f4ac396061d6b7320f7ae (table test-table [id=6d058d3e92d2476bb9470532e31da4ad]) successfully deleted
I20260811 12:34:15.642294 17969 ts_tablet_manager.cc:1905] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:15.649852 17969 ts_tablet_manager.cc:1918] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:15.650066 17969 log.cc:1196] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/c97b7d5cc83f4ac396061d6b7320f7ae
I20260811 12:34:15.650655 17969 ts_tablet_manager.cc:1939] T c97b7d5cc83f4ac396061d6b7320f7ae P 72d6e43777814bc39a42b13d33e58c5a: Deleting consensus metadata
I20260811 12:34:15.652567 17011 catalog_manager.cc:4841] TS 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507): tablet c97b7d5cc83f4ac396061d6b7320f7ae (table test-table [id=6d058d3e92d2476bb9470532e31da4ad]) successfully deleted
I20260811 12:34:15.653376 17919 tablet_bootstrap.cc:492] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: No bootstrap required, opened a new log
I20260811 12:34:15.653542 17919 ts_tablet_manager.cc:1397] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.005s
I20260811 12:34:15.654247 17919 raft_consensus.cc:348] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.654436 17919 raft_consensus.cc:374] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.654531 17919 raft_consensus.cc:729] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b3c5bbbbb246e4ac7e8946ca26ee0b, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.655087 17919 consensus_queue.cc:260] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.655689 17919 ts_tablet_manager.cc:1428] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 12:34:15.802570 17932 raft_consensus.cc:482] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:34:15.802819 17932 raft_consensus.cc:504] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.804523 17932 leader_election.cc:288] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507), 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391)
I20260811 12:34:15.812983 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.805210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:59933 (local address 127.3.204.65:37507)
0811 12:34:15.805411 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.805418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.806351 (+ 933us) server_negotiation.cc:407] Connection header received
0811 12:34:15.806506 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.806510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.806611 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.806701 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:15.807448 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.808549 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.810616 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.811771 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.811957 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.812044 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.812482 (+ 438us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.812642 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:34:15.813172 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.805167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:43045 (local address 127.3.204.66:38391)
0811 12:34:15.805361 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.805368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.806016 (+ 648us) server_negotiation.cc:407] Connection header received
0811 12:34:15.806350 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.806356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.806468 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.806572 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:15.807206 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.808324 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.810911 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.812168 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.812324 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.812404 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.812749 (+ 345us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.812890 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:34:15.813246 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.804858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:38391 (local address 127.3.204.67:43045)
0811 12:34:15.805888 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:15.805912 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:15.806153 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:15.806595 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:15.806600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:15.806624 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:15.807021 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.807032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.808488 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.808493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.810742 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.810758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.812410 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.812417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.812559 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.812583 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 12:34:15.812709 (+ 126us) client_negotiation.cc:240] Negotiation successful
0811 12:34:15.812846 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":48,"spinlock_wait_cycles":14080,"thread_start_us":1050,"threads_started":1}
I20260811 12:34:15.813990 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c29cf7743441bca552eef8151ef882" candidate_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" is_pre_election: true
I20260811 12:34:15.814240 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.804891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:37507 (local address 127.3.204.67:59933)
0811 12:34:15.806234 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:15.806249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:15.806380 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:15.806769 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:15.806773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:15.806828 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:15.807277 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.807290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.808677 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.808682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.810456 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.810471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.812004 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.812011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.812231 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.812270 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 12:34:15.813714 (+ 1444us) client_negotiation.cc:240] Negotiation successful
0811 12:34:15.813857 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":25,"spinlock_wait_cycles":1792,"thread_start_us":681,"threads_started":1}
I20260811 12:34:15.814371 17543 raft_consensus.cc:2454] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b89b55c1c404c7db3e7776c5cc7f266 in term 0.
I20260811 12:34:15.814916 17669 leader_election.cc:302] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b89b55c1c404c7db3e7776c5cc7f266, 43b3c5bbbbb246e4ac7e8946ca26ee0b; no voters:
I20260811 12:34:15.815325 17932 raft_consensus.cc:2781] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:34:15.815441 17932 raft_consensus.cc:482] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:34:15.815294 17314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c29cf7743441bca552eef8151ef882" candidate_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d6e43777814bc39a42b13d33e58c5a" is_pre_election: true
I20260811 12:34:15.815531 17932 raft_consensus.cc:3037] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:15.815757 17314 raft_consensus.cc:2454] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b89b55c1c404c7db3e7776c5cc7f266 in term 0.
I20260811 12:34:15.818470 17932 raft_consensus.cc:504] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.819130 17932 leader_election.cc:288] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [CANDIDATE]: Term 1 election: Requested vote from peers 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507), 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391)
I20260811 12:34:15.819568 17314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c29cf7743441bca552eef8151ef882" candidate_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d6e43777814bc39a42b13d33e58c5a"
I20260811 12:34:15.819725 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c29cf7743441bca552eef8151ef882" candidate_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b"
I20260811 12:34:15.819811 17314 raft_consensus.cc:3037] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:15.819964 17543 raft_consensus.cc:3037] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:15.823128 17543 raft_consensus.cc:2454] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b89b55c1c404c7db3e7776c5cc7f266 in term 1.
I20260811 12:34:15.823133 17314 raft_consensus.cc:2454] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b89b55c1c404c7db3e7776c5cc7f266 in term 1.
I20260811 12:34:15.823617 17669 leader_election.cc:302] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b89b55c1c404c7db3e7776c5cc7f266, 43b3c5bbbbb246e4ac7e8946ca26ee0b; no voters:
I20260811 12:34:15.823988 17932 raft_consensus.cc:2781] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:15.824527 17932 raft_consensus.cc:686] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 LEADER]: Becoming Leader. State: Replica: 2b89b55c1c404c7db3e7776c5cc7f266, State: Running, Role: LEADER
I20260811 12:34:15.824977 17932 consensus_queue.cc:237] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.829005 17034 catalog_manager.cc:5495] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } health_report { overall_health: HEALTHY } } }
Aug 11 12:34:15 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451652, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.126@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:15.902611 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.886639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.3.204.126:34375)
0811 12:34:15.887344 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:15.887354 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:15.887661 (+ 307us) server_negotiation.cc:407] Connection header received
0811 12:34:15.887808 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:15.887815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:15.887925 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:15.888066 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:15.889291 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.890578 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.892050 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.892732 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.892873 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.896632 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:15.896638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:15.896641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:15.896675 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:15.899410 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:15.900400 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:15.900404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:15.900406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:15.900541 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:15.900983 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:15.900987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:15.900989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:15.901267 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:15.901375 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:15.901963 (+ 588us) server_negotiation.cc:299] Negotiation successful
0811 12:34:15.902285 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":537,"thread_start_us":347,"threads_started":1}
I20260811 12:34:15.902643 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:15.886613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:34375 (local address 127.0.0.1:57178)
0811 12:34:15.887535 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:15.887558 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:15.887693 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:15.888181 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:15.888185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:15.888899 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:15.889120 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.889130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.890817 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.890822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.891879 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:15.891894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:15.892901 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:15.892905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:15.893058 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:15.893791 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:15.893809 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:15.896412 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:15.899604 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:15.899621 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:15.899629 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:15.900283 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:15.900657 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:15.900662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:15.900665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:15.900863 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:15.901406 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:15.901415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:15.901658 (+ 243us) client_negotiation.cc:769] Sending connection context
0811 12:34:15.901853 (+ 195us) client_negotiation.cc:240] Negotiation successful
0811 12:34:15.902172 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":581,"threads_started":1}
I20260811 12:34:15.908313 17034 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:57178:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 12:34:15.909416 17034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:34:15.916536 17511 tablet_service.cc:1467] Processing CreateTablet for tablet c5ecec4212c94c869f435df7848a3f01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51e9d045d88b49e3a1651c941cf63a85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:34:15.916625 17275 tablet_service.cc:1467] Processing CreateTablet for tablet c5ecec4212c94c869f435df7848a3f01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51e9d045d88b49e3a1651c941cf63a85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:34:15.917119 17511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ecec4212c94c869f435df7848a3f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.917402 17275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ecec4212c94c869f435df7848a3f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.919435 17749 tablet_service.cc:1467] Processing CreateTablet for tablet c5ecec4212c94c869f435df7848a3f01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51e9d045d88b49e3a1651c941cf63a85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 12:34:15.919940 17749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ecec4212c94c869f435df7848a3f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:15.925253 17921 tablet_bootstrap.cc:492] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: Bootstrap starting.
I20260811 12:34:15.926937 17921 tablet_bootstrap.cc:654] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.928109 17920 tablet_bootstrap.cc:492] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: Bootstrap starting.
I20260811 12:34:15.929996 17920 tablet_bootstrap.cc:654] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.930836 17919 tablet_bootstrap.cc:492] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Bootstrap starting.
I20260811 12:34:15.932799 17919 tablet_bootstrap.cc:654] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:15.934064 17921 tablet_bootstrap.cc:492] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: No bootstrap required, opened a new log
I20260811 12:34:15.934259 17921 ts_tablet_manager.cc:1397] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260811 12:34:15.935089 17921 raft_consensus.cc:348] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.935372 17921 raft_consensus.cc:374] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.935451 17921 raft_consensus.cc:729] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b89b55c1c404c7db3e7776c5cc7f266, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.935703 17921 consensus_queue.cc:260] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.936162 17932 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:15.936465 17932 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:34:15.938428 17921 ts_tablet_manager.cc:1428] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260811 12:34:15.940284 17920 tablet_bootstrap.cc:492] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: No bootstrap required, opened a new log
I20260811 12:34:15.940503 17920 ts_tablet_manager.cc:1397] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260811 12:34:15.941311 17920 raft_consensus.cc:348] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.941625 17920 raft_consensus.cc:374] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.941712 17920 raft_consensus.cc:729] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d6e43777814bc39a42b13d33e58c5a, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.941979 17920 consensus_queue.cc:260] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.943683 17935 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:15.944034 17935 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:34:15.945816 17920 ts_tablet_manager.cc:1428] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260811 12:34:15.947466 17919 tablet_bootstrap.cc:492] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: No bootstrap required, opened a new log
I20260811 12:34:15.947639 17919 ts_tablet_manager.cc:1397] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260811 12:34:15.948391 17919 raft_consensus.cc:348] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.948603 17919 raft_consensus.cc:374] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:15.948675 17919 raft_consensus.cc:729] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b3c5bbbbb246e4ac7e8946ca26ee0b, State: Initialized, Role: FOLLOWER
I20260811 12:34:15.948923 17919 consensus_queue.cc:260] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:15.949488 17919 ts_tablet_manager.cc:1428] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 12:34:15.949340 17929 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:15.949751 17929 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:34:16.061273 17932 raft_consensus.cc:482] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:34:16.061507 17932 raft_consensus.cc:504] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.062101 17932 leader_election.cc:288] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507), 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391)
I20260811 12:34:16.062711 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ecec4212c94c869f435df7848a3f01" candidate_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" is_pre_election: true
I20260811 12:34:16.062709 17314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ecec4212c94c869f435df7848a3f01" candidate_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d6e43777814bc39a42b13d33e58c5a" is_pre_election: true
I20260811 12:34:16.062958 17543 raft_consensus.cc:2454] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b89b55c1c404c7db3e7776c5cc7f266 in term 0.
I20260811 12:34:16.062964 17314 raft_consensus.cc:2454] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b89b55c1c404c7db3e7776c5cc7f266 in term 0.
I20260811 12:34:16.063413 17667 leader_election.cc:302] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b89b55c1c404c7db3e7776c5cc7f266, 72d6e43777814bc39a42b13d33e58c5a; no voters:
I20260811 12:34:16.063705 17932 raft_consensus.cc:2781] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:34:16.063794 17932 raft_consensus.cc:482] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:34:16.063923 17932 raft_consensus.cc:3037] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.066020 17932 raft_consensus.cc:504] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.066627 17932 leader_election.cc:288] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [CANDIDATE]: Term 1 election: Requested vote from peers 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507), 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391)
I20260811 12:34:16.067078 17314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ecec4212c94c869f435df7848a3f01" candidate_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d6e43777814bc39a42b13d33e58c5a"
I20260811 12:34:16.067169 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ecec4212c94c869f435df7848a3f01" candidate_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b"
I20260811 12:34:16.067299 17314 raft_consensus.cc:3037] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.067421 17543 raft_consensus.cc:3037] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.069514 17314 raft_consensus.cc:2454] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b89b55c1c404c7db3e7776c5cc7f266 in term 1.
I20260811 12:34:16.069731 17543 raft_consensus.cc:2454] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b89b55c1c404c7db3e7776c5cc7f266 in term 1.
I20260811 12:34:16.069959 17667 leader_election.cc:302] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b89b55c1c404c7db3e7776c5cc7f266, 72d6e43777814bc39a42b13d33e58c5a; no voters:
I20260811 12:34:16.070246 17932 raft_consensus.cc:2781] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:16.070417 17932 raft_consensus.cc:686] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 LEADER]: Becoming Leader. State: Replica: 2b89b55c1c404c7db3e7776c5cc7f266, State: Running, Role: LEADER
I20260811 12:34:16.070676 17932 consensus_queue.cc:237] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.072263 17978 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: TxnStatusTablet state changed. Reason: New leader 2b89b55c1c404c7db3e7776c5cc7f266. Latest consensus state: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:16.072646 17978 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:34:16.073338 18005 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 12:34:16.073593 18005 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 12:34:16.074028 17034 catalog_manager.cc:5495] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } health_report { overall_health: HEALTHY } } }
I20260811 12:34:16.106153 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.099095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:39335 (local address 127.0.0.1:51074)
0811 12:34:16.099370 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:16.099395 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:16.099581 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:16.100266 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:16.100269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:16.100280 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:34:16.100460 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.100469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.102100 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.102108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.103184 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.103200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.104268 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.104273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.104431 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.104503 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.105692 (+ 1189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.105704 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 12:34:16.105754 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 12:34:16.105886 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260811 12:34:16.106297 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.099134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.3.204.67:39335)
0811 12:34:16.099777 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.099784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.099805 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:34:16.099918 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.099924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.100041 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.100253 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:16.100599 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.101910 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.103348 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.104115 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.104253 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.104713 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.105596 (+ 883us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.105749 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.105829 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.105923 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":324,"threads_started":1}
I20260811 12:34:16.111135 17543 raft_consensus.cc:1261] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Refusing update from remote peer 2b89b55c1c404c7db3e7776c5cc7f266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:16.111253 17314 raft_consensus.cc:1261] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Refusing update from remote peer 2b89b55c1c404c7db3e7776c5cc7f266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:16.111920 17978 consensus_queue.cc:1035] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:16.112609 17932 consensus_queue.cc:1035] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:16.117595 17935 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: TxnStatusTablet state changed. Reason: New leader 2b89b55c1c404c7db3e7776c5cc7f266. Latest consensus state: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:16.117887 17935 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:34:16.121436 18016 mvcc.cc:205] Tried to move back new op lower bound from 7317305983423676416 to 7317305983267684352. Current Snapshot: MvccSnapshot[applied={T|T < 7317305983423676416}]
I20260811 12:34:16.124768 18010 mvcc.cc:205] Tried to move back new op lower bound from 7317305983423676416 to 7317305983267684352. Current Snapshot: MvccSnapshot[applied={T|T < 7317305983423676416}]
I20260811 12:34:16.128784 17929 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: TxnStatusTablet state changed. Reason: New leader 2b89b55c1c404c7db3e7776c5cc7f266. Latest consensus state: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:16.129161 17929 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:34:16.128914 17978 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:16.130092 17978 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:34:16.134482 17935 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:16.134791 17935 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 72d6e43777814bc39a42b13d33e58c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:34:16.136607 18021 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:16.137149 18021 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266: This TxnStatusTablet replica's current role is: LEADER
I20260811 12:34:16.141587 17929 tablet_replica.cc:439] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } } }
I20260811 12:34:16.141860 17929 tablet_replica.cc:442] T c5ecec4212c94c869f435df7848a3f01 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 12:34:16.151305 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.145161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.3.204.67:39335)
0811 12:34:16.145380 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.145386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.145788 (+ 402us) server_negotiation.cc:407] Connection header received
0811 12:34:16.145986 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.145991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.146056 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.146136 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:16.146694 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.147549 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.149512 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.149947 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.150041 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.150572 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.150689 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.150763 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.150926 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.150999 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:34:16.168953 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.163359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:45073 (local address 127.3.204.67:39335)
0811 12:34:16.163600 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.163608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.164019 (+ 411us) server_negotiation.cc:407] Connection header received
0811 12:34:16.164233 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.164236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.164326 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.164409 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:16.164880 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.165645 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.167150 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.167574 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.167679 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.168179 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.168296 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.168374 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.168498 (+ 124us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.168575 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 12:34:16.169056 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.163219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:39335 (local address 127.3.204.67:45073)
0811 12:34:16.163913 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:16.163928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:16.164106 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:16.164429 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:16.164433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:16.164444 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:34:16.164737 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.164746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.165785 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.165789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.167015 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.167028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.167754 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.167759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.167894 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.167954 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.168388 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.168399 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 12:34:16.168468 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 12:34:16.168617 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":1,"thread_start_us":405,"threads_started":1}
I20260811 12:34:16.178805 17312 raft_consensus.cc:1261] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Refusing update from remote peer 2b89b55c1c404c7db3e7776c5cc7f266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:16.178795 17543 raft_consensus.cc:1261] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Refusing update from remote peer 2b89b55c1c404c7db3e7776c5cc7f266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:16.179360 18021 consensus_queue.cc:1035] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:16.179845 17978 consensus_queue.cc:1035] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:16.193202 18013 mvcc.cc:205] Tried to move back new op lower bound from 7317305983702355968 to 7317305982263762944. Current Snapshot: MvccSnapshot[applied={T|T < 7317305983702355968}]
I20260811 12:34:16.249138 17034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:16.249392 17034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:16.252663 17034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: Sending DeleteTablet for 3 replicas of tablet 40c29cf7743441bca552eef8151ef882
I20260811 12:34:16.253546 17275 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c29cf7743441bca552eef8151ef882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:42010
I20260811 12:34:16.253579 17511 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c29cf7743441bca552eef8151ef882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:41816
I20260811 12:34:16.254220 17748 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c29cf7743441bca552eef8151ef882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:51070
I20260811 12:34:16.254779 18043 tablet_replica.cc:330] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: stopping tablet replica
I20260811 12:34:16.255003 18043 raft_consensus.cc:2229] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:16.255259 18043 raft_consensus.cc:2258] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.255764 18046 tablet_replica.cc:330] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: stopping tablet replica
I20260811 12:34:16.255967 18046 raft_consensus.cc:2229] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:16.256283 18046 raft_consensus.cc:2258] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.257457 18046 ts_tablet_manager.cc:1905] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.263587 18047 tablet_replica.cc:330] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: stopping tablet replica
I20260811 12:34:16.263972 18047 raft_consensus.cc:2229] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:34:16.264712 18047 raft_consensus.cc:2258] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.264812 18046 ts_tablet_manager.cc:1918] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:34:16.264966 18046 log.cc:1196] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/40c29cf7743441bca552eef8151ef882
I20260811 12:34:16.265002 18043 ts_tablet_manager.cc:1905] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.265450 18046 ts_tablet_manager.cc:1939] T 40c29cf7743441bca552eef8151ef882 P 72d6e43777814bc39a42b13d33e58c5a: Deleting consensus metadata
I20260811 12:34:16.266126 18047 ts_tablet_manager.cc:1905] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.266619 17011 catalog_manager.cc:4841] TS 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507): tablet 40c29cf7743441bca552eef8151ef882 (table test-table [id=e01baa9f954449129da56a01eff4a366]) successfully deleted
I20260811 12:34:16.270586 18043 ts_tablet_manager.cc:1918] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:34:16.270713 18043 log.cc:1196] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/40c29cf7743441bca552eef8151ef882
I20260811 12:34:16.271119 18043 ts_tablet_manager.cc:1939] T 40c29cf7743441bca552eef8151ef882 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting consensus metadata
I20260811 12:34:16.273778 18047 ts_tablet_manager.cc:1918] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:34:16.273921 18047 log.cc:1196] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/40c29cf7743441bca552eef8151ef882
I20260811 12:34:16.274353 18047 ts_tablet_manager.cc:1939] T 40c29cf7743441bca552eef8151ef882 P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting consensus metadata
I20260811 12:34:16.275666 17011 catalog_manager.cc:4841] TS 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335): tablet 40c29cf7743441bca552eef8151ef882 (table test-table [id=e01baa9f954449129da56a01eff4a366]) successfully deleted
I20260811 12:34:16.276518 17012 catalog_manager.cc:4841] TS 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391): tablet 40c29cf7743441bca552eef8151ef882 (table test-table [id=e01baa9f954449129da56a01eff4a366]) successfully deleted
Aug 11 12:34:16 dist-test-slave-lgr5 krb5kdc[16960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 12:34:16 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:16 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 12:34:16.330813 17992 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:16.317618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.3.204.126:34375)
0811 12:34:16.317849 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.317856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.318372 (+ 516us) server_negotiation.cc:407] Connection header received
0811 12:34:16.318561 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.318565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.318637 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.318726 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:16.320631 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.321474 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.323353 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.323774 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.323889 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.330560 (+ 6671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57188: BlockingRecv error: recv got EOF from 127.0.0.1:57188 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 11 12:34:16 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM
I20260811 12:34:16.363093 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.339143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.3.204.126:34375)
0811 12:34:16.339322 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.339329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.346435 (+ 7106us) server_negotiation.cc:407] Connection header received
0811 12:34:16.346617 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.346623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.346712 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.346829 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:16.348578 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.349800 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.351739 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.352234 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.352343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.357218 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:16.357224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:16.357228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:16.357295 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:16.360241 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:16.361108 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:16.361113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:16.361115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:16.361289 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:16.361721 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:16.361727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:16.361730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:16.362035 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:16.362250 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.362596 (+ 346us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.362810 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:34:16.367266 17034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:34:16.368040 17034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:34:16.375770 17275 tablet_service.cc:1467] Processing CreateTablet for tablet 46bf0b3f25164198bdb2e0ce9263ca53 (DEFAULT_TABLE table=test-table [id=6bf8e6d64e5049699855489765653daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:16.376363 17275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bf0b3f25164198bdb2e0ce9263ca53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:16.376295 17511 tablet_service.cc:1467] Processing CreateTablet for tablet 46bf0b3f25164198bdb2e0ce9263ca53 (DEFAULT_TABLE table=test-table [id=6bf8e6d64e5049699855489765653daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:16.376766 17511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bf0b3f25164198bdb2e0ce9263ca53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:16.376881 17748 tablet_service.cc:1467] Processing CreateTablet for tablet 46bf0b3f25164198bdb2e0ce9263ca53 (DEFAULT_TABLE table=test-table [id=6bf8e6d64e5049699855489765653daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:16.377424 17748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bf0b3f25164198bdb2e0ce9263ca53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:16.383170 17920 tablet_bootstrap.cc:492] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: Bootstrap starting.
I20260811 12:34:16.383396 17919 tablet_bootstrap.cc:492] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Bootstrap starting.
I20260811 12:34:16.384986 17920 tablet_bootstrap.cc:654] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:16.384986 17919 tablet_bootstrap.cc:654] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:16.386106 17921 tablet_bootstrap.cc:492] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: Bootstrap starting.
I20260811 12:34:16.388043 17921 tablet_bootstrap.cc:654] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:16.400580 17921 tablet_bootstrap.cc:492] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: No bootstrap required, opened a new log
I20260811 12:34:16.400810 17921 ts_tablet_manager.cc:1397] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260811 12:34:16.400848 17920 tablet_bootstrap.cc:492] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: No bootstrap required, opened a new log
I20260811 12:34:16.401193 17920 ts_tablet_manager.cc:1397] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.005s
I20260811 12:34:16.401798 17921 raft_consensus.cc:348] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.402125 17921 raft_consensus.cc:374] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:16.402235 17921 raft_consensus.cc:729] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b89b55c1c404c7db3e7776c5cc7f266, State: Initialized, Role: FOLLOWER
I20260811 12:34:16.402312 17920 raft_consensus.cc:348] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.402592 17921 consensus_queue.cc:260] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.402755 17920 raft_consensus.cc:374] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:16.403112 17920 raft_consensus.cc:729] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d6e43777814bc39a42b13d33e58c5a, State: Initialized, Role: FOLLOWER
I20260811 12:34:16.403487 17920 consensus_queue.cc:260] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.404194 17920 ts_tablet_manager.cc:1428] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260811 12:34:16.406177 17921 ts_tablet_manager.cc:1428] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260811 12:34:16.407163 17919 tablet_bootstrap.cc:492] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: No bootstrap required, opened a new log
I20260811 12:34:16.407341 17919 ts_tablet_manager.cc:1397] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20260811 12:34:16.408277 17919 raft_consensus.cc:348] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.408548 17919 raft_consensus.cc:374] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:16.408644 17919 raft_consensus.cc:729] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b3c5bbbbb246e4ac7e8946ca26ee0b, State: Initialized, Role: FOLLOWER
I20260811 12:34:16.408898 17919 consensus_queue.cc:260] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.409457 17919 ts_tablet_manager.cc:1428] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 12:34:16.577865 17929 raft_consensus.cc:482] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:34:16.578105 17929 raft_consensus.cc:504] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.579655 17929 leader_election.cc:288] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507), 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335)
I20260811 12:34:16.588480 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.580166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:34163 (local address 127.3.204.65:37507)
0811 12:34:16.582239 (+ 2073us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.582244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.582263 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:34:16.582337 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.582342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.582415 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.583029 (+ 614us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:16.583203 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.583979 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.586144 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.587393 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.587576 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.587655 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.587978 (+ 323us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.588159 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1957,"thread_start_us":1860,"threads_started":1}
I20260811 12:34:16.588460 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.579985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:37507 (local address 127.3.204.66:34163)
0811 12:34:16.580641 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:16.580670 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:16.580899 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:16.582548 (+ 1649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:16.582552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:16.582581 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:16.582842 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.582855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.584138 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.584143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.585953 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.585968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.587605 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.587611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.587800 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.587838 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 12:34:16.587933 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 12:34:16.588112 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":325,"threads_started":1}
I20260811 12:34:16.588717 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.581085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:48279 (local address 127.3.204.67:39335)
0811 12:34:16.581258 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.581264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.581788 (+ 524us) server_negotiation.cc:407] Connection header received
0811 12:34:16.582563 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.582569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.582653 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.582749 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:16.583361 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.584542 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.586638 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.587536 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.587705 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.587784 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.588280 (+ 496us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.588427 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 12:34:16.589520 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.580748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:39335 (local address 127.3.204.66:48279)
0811 12:34:16.581692 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:16.581710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:16.582385 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:16.582783 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:16.582787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:16.582809 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:16.583148 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.583186 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.584706 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.584711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.586485 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.586496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.587844 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.587851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.588037 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.588073 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:34:16.588969 (+ 896us) client_negotiation.cc:240] Negotiation successful
0811 12:34:16.589118 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":2,"thread_start_us":1358,"threads_started":1}
I20260811 12:34:16.589498 17312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bf0b3f25164198bdb2e0ce9263ca53" candidate_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d6e43777814bc39a42b13d33e58c5a" is_pre_election: true
I20260811 12:34:16.589793 17312 raft_consensus.cc:2454] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43b3c5bbbbb246e4ac7e8946ca26ee0b in term 0.
I20260811 12:34:16.590109 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bf0b3f25164198bdb2e0ce9263ca53" candidate_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" is_pre_election: true
I20260811 12:34:16.590261 17436 leader_election.cc:302] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b3c5bbbbb246e4ac7e8946ca26ee0b, 72d6e43777814bc39a42b13d33e58c5a; no voters:
I20260811 12:34:16.590377 17787 raft_consensus.cc:2454] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43b3c5bbbbb246e4ac7e8946ca26ee0b in term 0.
I20260811 12:34:16.590703 17929 raft_consensus.cc:2781] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:34:16.590830 17929 raft_consensus.cc:482] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:34:16.590975 17929 raft_consensus.cc:3037] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.593171 17929 raft_consensus.cc:504] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.593696 17929 leader_election.cc:288] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [CANDIDATE]: Term 1 election: Requested vote from peers 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507), 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335)
I20260811 12:34:16.594151 17312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bf0b3f25164198bdb2e0ce9263ca53" candidate_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d6e43777814bc39a42b13d33e58c5a"
I20260811 12:34:16.594359 17312 raft_consensus.cc:3037] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.594296 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bf0b3f25164198bdb2e0ce9263ca53" candidate_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b89b55c1c404c7db3e7776c5cc7f266"
I20260811 12:34:16.594514 17787 raft_consensus.cc:3037] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.596570 17312 raft_consensus.cc:2454] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43b3c5bbbbb246e4ac7e8946ca26ee0b in term 1.
I20260811 12:34:16.596949 17436 leader_election.cc:302] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b3c5bbbbb246e4ac7e8946ca26ee0b, 72d6e43777814bc39a42b13d33e58c5a; no voters:
I20260811 12:34:16.597214 17929 raft_consensus.cc:2781] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:16.597677 17929 raft_consensus.cc:686] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 LEADER]: Becoming Leader. State: Replica: 43b3c5bbbbb246e4ac7e8946ca26ee0b, State: Running, Role: LEADER
I20260811 12:34:16.597680 17787 raft_consensus.cc:2454] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43b3c5bbbbb246e4ac7e8946ca26ee0b in term 1.
I20260811 12:34:16.598093 17929 consensus_queue.cc:237] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.601856 17035 catalog_manager.cc:5495] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66). New cstate: current_term: 1 leader_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } health_report { overall_health: UNKNOWN } } }
I20260811 12:34:16.641470 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.634296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.3.204.66:38391)
0811 12:34:16.635098 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.635102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.635122 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:34:16.635217 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.635221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.635303 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.635458 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:16.635946 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.636872 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.638850 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.639321 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.639436 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.639994 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.640769 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.640894 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.641022 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.641098 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":497,"threads_started":1}
I20260811 12:34:16.645504 17787 raft_consensus.cc:1261] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Refusing update from remote peer 43b3c5bbbbb246e4ac7e8946ca26ee0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:16.645617 17312 raft_consensus.cc:1261] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Refusing update from remote peer 43b3c5bbbbb246e4ac7e8946ca26ee0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:16.646322 18081 consensus_queue.cc:1035] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:16.646715 17929 consensus_queue.cc:1035] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:16.674542 17035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:16.674846 17035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:16.678785 17035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: Sending DeleteTablet for 3 replicas of tablet 46bf0b3f25164198bdb2e0ce9263ca53
I20260811 12:34:16.679787 17511 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bf0b3f25164198bdb2e0ce9263ca53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:41816
I20260811 12:34:16.680181 17275 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bf0b3f25164198bdb2e0ce9263ca53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:42010
I20260811 12:34:16.680294 18043 tablet_replica.cc:330] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: stopping tablet replica
I20260811 12:34:16.680482 18046 tablet_replica.cc:330] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: stopping tablet replica
I20260811 12:34:16.680552 18043 raft_consensus.cc:2229] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:34:16.680699 18046 raft_consensus.cc:2229] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:16.680949 18046 raft_consensus.cc:2258] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.681161 18043 raft_consensus.cc:2258] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.682464 18046 ts_tablet_manager.cc:1905] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.684443 17748 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bf0b3f25164198bdb2e0ce9263ca53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:51070
I20260811 12:34:16.684775 18047 tablet_replica.cc:330] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: stopping tablet replica
I20260811 12:34:16.684978 18047 raft_consensus.cc:2229] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:16.685225 18047 raft_consensus.cc:2258] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.685650 18043 ts_tablet_manager.cc:1905] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.691092 18047 ts_tablet_manager.cc:1905] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.693130 18043 ts_tablet_manager.cc:1918] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:16.693382 18043 log.cc:1196] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/46bf0b3f25164198bdb2e0ce9263ca53
I20260811 12:34:16.693957 18043 ts_tablet_manager.cc:1939] T 46bf0b3f25164198bdb2e0ce9263ca53 P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting consensus metadata
I20260811 12:34:16.695436 17012 catalog_manager.cc:4841] TS 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391): tablet 46bf0b3f25164198bdb2e0ce9263ca53 (table test-table [id=6bf8e6d64e5049699855489765653daa]) successfully deleted
I20260811 12:34:16.696767 18046 ts_tablet_manager.cc:1918] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:16.697026 18046 log.cc:1196] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/46bf0b3f25164198bdb2e0ce9263ca53
I20260811 12:34:16.697501 18046 ts_tablet_manager.cc:1939] T 46bf0b3f25164198bdb2e0ce9263ca53 P 72d6e43777814bc39a42b13d33e58c5a: Deleting consensus metadata
I20260811 12:34:16.698012 17035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 12:34:16.699647 17011 catalog_manager.cc:4841] TS 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507): tablet 46bf0b3f25164198bdb2e0ce9263ca53 (table test-table [id=6bf8e6d64e5049699855489765653daa]) successfully deleted
W20260811 12:34:16.700388 17035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:34:16.700883 18047 ts_tablet_manager.cc:1918] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:16.701052 18047 log.cc:1196] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/46bf0b3f25164198bdb2e0ce9263ca53
I20260811 12:34:16.701490 18047 ts_tablet_manager.cc:1939] T 46bf0b3f25164198bdb2e0ce9263ca53 P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting consensus metadata
I20260811 12:34:16.703605 17011 catalog_manager.cc:4841] TS 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335): tablet 46bf0b3f25164198bdb2e0ce9263ca53 (table test-table [id=6bf8e6d64e5049699855489765653daa]) successfully deleted
I20260811 12:34:16.708943 17511 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8243daeb22404aac37fc7005bd792d (DEFAULT_TABLE table=test-table [id=cfe1a18ea0d049c5979efdf8f4104e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:16.709475 17511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e8243daeb22404aac37fc7005bd792d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:16.709470 17748 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8243daeb22404aac37fc7005bd792d (DEFAULT_TABLE table=test-table [id=cfe1a18ea0d049c5979efdf8f4104e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:16.709910 17748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e8243daeb22404aac37fc7005bd792d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:16.714901 17275 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8243daeb22404aac37fc7005bd792d (DEFAULT_TABLE table=test-table [id=cfe1a18ea0d049c5979efdf8f4104e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:16.715363 17275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e8243daeb22404aac37fc7005bd792d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:16.718101 17921 tablet_bootstrap.cc:492] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: Bootstrap starting.
I20260811 12:34:16.721215 17921 tablet_bootstrap.cc:654] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:16.722754 17920 tablet_bootstrap.cc:492] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: Bootstrap starting.
I20260811 12:34:16.723907 17919 tablet_bootstrap.cc:492] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Bootstrap starting.
I20260811 12:34:16.724478 17920 tablet_bootstrap.cc:654] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:16.725678 17919 tablet_bootstrap.cc:654] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:16.737803 17920 tablet_bootstrap.cc:492] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: No bootstrap required, opened a new log
I20260811 12:34:16.738413 17920 ts_tablet_manager.cc:1397] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260811 12:34:16.739199 17920 raft_consensus.cc:348] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.741127 17920 raft_consensus.cc:374] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:16.741217 17920 raft_consensus.cc:729] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d6e43777814bc39a42b13d33e58c5a, State: Initialized, Role: FOLLOWER
I20260811 12:34:16.741506 17920 consensus_queue.cc:260] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.742062 17920 ts_tablet_manager.cc:1428] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260811 12:34:16.744171 17921 tablet_bootstrap.cc:492] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: No bootstrap required, opened a new log
I20260811 12:34:16.744387 17921 ts_tablet_manager.cc:1397] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.017s
I20260811 12:34:16.745267 17921 raft_consensus.cc:348] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.745563 17921 raft_consensus.cc:374] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:16.745649 17921 raft_consensus.cc:729] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b89b55c1c404c7db3e7776c5cc7f266, State: Initialized, Role: FOLLOWER
I20260811 12:34:16.745987 17921 consensus_queue.cc:260] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.746614 17919 tablet_bootstrap.cc:492] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: No bootstrap required, opened a new log
I20260811 12:34:16.746670 17921 ts_tablet_manager.cc:1428] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 12:34:16.746977 17919 ts_tablet_manager.cc:1397] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.000s
I20260811 12:34:16.747854 17919 raft_consensus.cc:348] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.748201 17919 raft_consensus.cc:374] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:16.748298 17919 raft_consensus.cc:729] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b3c5bbbbb246e4ac7e8946ca26ee0b, State: Initialized, Role: FOLLOWER
I20260811 12:34:16.748543 17919 consensus_queue.cc:260] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.749119 17919 ts_tablet_manager.cc:1428] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 12:34:16.758787 17935 raft_consensus.cc:482] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:34:16.758987 17935 raft_consensus.cc:504] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.759732 17935 leader_election.cc:288] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391), 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335)
I20260811 12:34:16.760231 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e8243daeb22404aac37fc7005bd792d" candidate_uuid: "72d6e43777814bc39a42b13d33e58c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" is_pre_election: true
I20260811 12:34:16.760486 17543 raft_consensus.cc:2454] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d6e43777814bc39a42b13d33e58c5a in term 0.
I20260811 12:34:16.760488 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e8243daeb22404aac37fc7005bd792d" candidate_uuid: "72d6e43777814bc39a42b13d33e58c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" is_pre_election: true
I20260811 12:34:16.760746 17787 raft_consensus.cc:2454] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d6e43777814bc39a42b13d33e58c5a in term 0.
I20260811 12:34:16.761101 17194 leader_election.cc:302] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b3c5bbbbb246e4ac7e8946ca26ee0b, 72d6e43777814bc39a42b13d33e58c5a; no voters:
I20260811 12:34:16.761390 17935 raft_consensus.cc:2781] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:34:16.761524 17935 raft_consensus.cc:482] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:34:16.761606 17935 raft_consensus.cc:3037] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.764668 17935 raft_consensus.cc:504] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.765792 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e8243daeb22404aac37fc7005bd792d" candidate_uuid: "72d6e43777814bc39a42b13d33e58c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b"
I20260811 12:34:16.765831 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e8243daeb22404aac37fc7005bd792d" candidate_uuid: "72d6e43777814bc39a42b13d33e58c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b89b55c1c404c7db3e7776c5cc7f266"
I20260811 12:34:16.766038 17543 raft_consensus.cc:3037] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.766083 17787 raft_consensus.cc:3037] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:16.769511 17543 raft_consensus.cc:2454] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d6e43777814bc39a42b13d33e58c5a in term 1.
I20260811 12:34:16.769517 17787 raft_consensus.cc:2454] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d6e43777814bc39a42b13d33e58c5a in term 1.
I20260811 12:34:16.769989 17192 leader_election.cc:302] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b89b55c1c404c7db3e7776c5cc7f266, 72d6e43777814bc39a42b13d33e58c5a; no voters:
I20260811 12:34:16.770607 17935 leader_election.cc:288] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [CANDIDATE]: Term 1 election: Requested vote from peers 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391), 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335)
I20260811 12:34:16.770767 18105 raft_consensus.cc:2781] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:16.770973 18105 raft_consensus.cc:686] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 1 LEADER]: Becoming Leader. State: Replica: 72d6e43777814bc39a42b13d33e58c5a, State: Running, Role: LEADER
I20260811 12:34:16.771344 18105 consensus_queue.cc:237] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } }
I20260811 12:34:16.775076 17034 catalog_manager.cc:5495] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65). New cstate: current_term: 1 leader_uuid: "72d6e43777814bc39a42b13d33e58c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d6e43777814bc39a42b13d33e58c5a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 37507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 } health_report { overall_health: UNKNOWN } } }
I20260811 12:34:16.807611 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.799957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.3.204.65:37507)
0811 12:34:16.800161 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.800167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.800187 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:34:16.800253 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.800258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.800362 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.800458 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:16.801444 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.802602 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.805032 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.805700 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.805824 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.806463 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.806608 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.806689 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.807217 (+ 528us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.807303 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 12:34:16.842984 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.832938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:43635 (local address 127.3.204.67:39335)
0811 12:34:16.833126 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.833136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.836113 (+ 2977us) server_negotiation.cc:407] Connection header received
0811 12:34:16.836506 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.836512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.836607 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.836702 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:16.837448 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.838617 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.840698 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.841466 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.841600 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.842153 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.842303 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.842399 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.842555 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.842654 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 12:34:16.843294 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.832915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:39335 (local address 127.3.204.65:43635)
0811 12:34:16.835953 (+ 3038us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:16.835976 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:16.836202 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:16.836879 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:16.836884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:16.836911 (+ 27us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:34:16.837241 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.837254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.838776 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.838782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.840541 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.840557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.841666 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.841673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.841856 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.841948 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.842408 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.842420 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 12:34:16.842502 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 12:34:16.842657 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2827,"thread_start_us":2672,"threads_started":1}
I20260811 12:34:16.855664 17545 raft_consensus.cc:1261] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Refusing update from remote peer 72d6e43777814bc39a42b13d33e58c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:16.856827 17935 consensus_queue.cc:1035] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43b3c5bbbbb246e4ac7e8946ca26ee0b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:16.858067 17787 raft_consensus.cc:1261] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Refusing update from remote peer 72d6e43777814bc39a42b13d33e58c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:16.861287 17935 consensus_queue.cc:1035] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b89b55c1c404c7db3e7776c5cc7f266" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:16.926283 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.913536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:47193 (local address 127.3.204.65:37507)
0811 12:34:16.913697 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.913702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:16.919736 (+ 6034us) server_negotiation.cc:407] Connection header received
0811 12:34:16.920068 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:16.920074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:16.920190 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:16.920253 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:16.921322 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.922631 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.924577 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.924982 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.925094 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.925547 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.925653 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.925724 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:16.925871 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 12:34:16.925970 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 12:34:16.926406 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.913457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:37507 (local address 127.3.204.67:47193)
0811 12:34:16.919596 (+ 6139us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:16.919646 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:16.919864 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:16.920550 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:16.920577 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:16.920594 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 12:34:16.921119 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.921135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.922807 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.922811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.924438 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:16.924451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:16.925178 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:16.925184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:16.925353 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:16.925395 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 12:34:16.925758 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 12:34:16.925769 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 12:34:16.925838 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 12:34:16.926024 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5853,"thread_start_us":10561,"threads_started":1}
I20260811 12:34:16.973145 17034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:16.973421 17034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:16.977381 17034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cff04494a32b4672a5be284cb5d06e7a: Sending DeleteTablet for 3 replicas of tablet 2e8243daeb22404aac37fc7005bd792d
I20260811 12:34:16.978801 17748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e8243daeb22404aac37fc7005bd792d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:51070
I20260811 12:34:16.979157 17511 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e8243daeb22404aac37fc7005bd792d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:41816
I20260811 12:34:16.979485 18047 tablet_replica.cc:330] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: stopping tablet replica
I20260811 12:34:16.979687 18047 raft_consensus.cc:2229] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:16.979895 18047 raft_consensus.cc:2258] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.980206 18043 tablet_replica.cc:330] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: stopping tablet replica
I20260811 12:34:16.980405 18043 raft_consensus.cc:2229] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:16.980631 18043 raft_consensus.cc:2258] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.981978 18043 ts_tablet_manager.cc:1905] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.982050 17275 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e8243daeb22404aac37fc7005bd792d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:16 UTC) from {username='oryx', principal='oryx/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:42010
I20260811 12:34:16.982348 18046 tablet_replica.cc:330] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: stopping tablet replica
I20260811 12:34:16.982527 18046 raft_consensus.cc:2229] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:34:16.982862 18046 raft_consensus.cc:2258] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:16.984310 18047 ts_tablet_manager.cc:1905] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.988965 18043 ts_tablet_manager.cc:1918] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:34:16.989125 18043 log.cc:1196] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/2e8243daeb22404aac37fc7005bd792d
I20260811 12:34:16.989125 18046 ts_tablet_manager.cc:1905] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:16.989575 18043 ts_tablet_manager.cc:1939] T 2e8243daeb22404aac37fc7005bd792d P 43b3c5bbbbb246e4ac7e8946ca26ee0b: Deleting consensus metadata
I20260811 12:34:16.990906 17012 catalog_manager.cc:4841] TS 43b3c5bbbbb246e4ac7e8946ca26ee0b (127.3.204.66:38391): tablet 2e8243daeb22404aac37fc7005bd792d (table test-table [id=cfe1a18ea0d049c5979efdf8f4104e4b]) successfully deleted
I20260811 12:34:16.993471 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 17160
I20260811 12:34:16.996891 18047 ts_tablet_manager.cc:1918] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:34:16.999699 18046 ts_tablet_manager.cc:1918] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 12:34:16.999935 18046 log.cc:1196] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/2e8243daeb22404aac37fc7005bd792d
I20260811 12:34:17.000444 18046 ts_tablet_manager.cc:1939] T 2e8243daeb22404aac37fc7005bd792d P 72d6e43777814bc39a42b13d33e58c5a: Deleting consensus metadata
I20260811 12:34:17.000653 18047 log.cc:1196] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/2e8243daeb22404aac37fc7005bd792d
I20260811 12:34:17.001204 18047 ts_tablet_manager.cc:1939] T 2e8243daeb22404aac37fc7005bd792d P 2b89b55c1c404c7db3e7776c5cc7f266: Deleting consensus metadata
I20260811 12:34:17.002687 17011 catalog_manager.cc:4841] TS 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507): tablet 2e8243daeb22404aac37fc7005bd792d (table test-table [id=cfe1a18ea0d049c5979efdf8f4104e4b]) successfully deleted
I20260811 12:34:17.003906 17011 catalog_manager.cc:4841] TS 2b89b55c1c404c7db3e7776c5cc7f266 (127.3.204.67:39335): tablet 2e8243daeb22404aac37fc7005bd792d (table test-table [id=cfe1a18ea0d049c5979efdf8f4104e4b]) successfully deleted
Aug 11 12:34:17 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.65@KRBTEST.COM
I20260811 12:34:17.028498 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:16.995087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.3.204.65:37507)
0811 12:34:16.995270 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:16.995277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:17.009122 (+ 13845us) server_negotiation.cc:407] Connection header received
0811 12:34:17.009305 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:17.009311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:17.009418 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:17.009533 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:17.011359 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:17.012357 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:17.014298 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:17.015006 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:17.015107 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:17.020672 (+ 5565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:17.020696 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:17.020718 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:17.020777 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:17.024935 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:17.025831 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:17.025841 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:17.025845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:17.026015 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:17.026435 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:17.026440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:17.026443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:17.026990 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:17.027292 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:17.027592 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 12:34:17.027844 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 12:34:17.029313 17388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:17.201262 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 17160
I20260811 12:34:17.240248 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 17408
Aug 11 12:34:17 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.66@KRBTEST.COM
I20260811 12:34:17.257362 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:17.241091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.3.204.66:38391)
0811 12:34:17.241686 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:17.241691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:17.241708 (+ 17us) server_negotiation.cc:407] Connection header received
0811 12:34:17.241767 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:17.241770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:17.241842 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:17.241932 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:17.243511 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:17.244443 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:17.246219 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:17.246692 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:17.246802 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:17.251693 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:17.251706 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:17.251721 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:17.251755 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:17.254634 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:17.255284 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:17.255291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:17.255294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:17.255437 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:17.255786 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:17.255790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:17.255792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:17.256227 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:17.256524 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:17.256804 (+ 280us) server_negotiation.cc:299] Negotiation successful
0811 12:34:17.257039 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":305,"threads_started":1}
I20260811 12:34:17.257992 17624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:17.387212 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 17408
I20260811 12:34:17.417245 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 17640
Aug 11 12:34:17 dist-test-slave-lgr5 krb5kdc[16960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.204.67@KRBTEST.COM
I20260811 12:34:17.431670 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:17.430901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:37507 (local address 127.3.204.67:44779)
0811 12:34:17.431372 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:17.431475 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":251,"threads_started":1}
W20260811 12:34:17.431965 17667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.65:37507: connect: Connection refused (error 111)
I20260811 12:34:17.433383 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:17.418189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.3.204.67:39335)
0811 12:34:17.418668 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:17.418674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:17.418691 (+ 17us) server_negotiation.cc:407] Connection header received
0811 12:34:17.418747 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:17.418750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:17.418808 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:17.418909 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:17.420428 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:17.421194 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:17.422427 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:17.422827 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:17.422920 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:17.427493 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:17.427511 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:17.427529 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:17.427566 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:17.430361 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:17.431054 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:17.431062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:17.431065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:17.431239 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:17.431630 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:17.431634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:17.431636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:17.432166 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:17.432503 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:17.432807 (+ 304us) server_negotiation.cc:299] Negotiation successful
0811 12:34:17.433003 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":275,"threads_started":1}
I20260811 12:34:17.433645 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:17.433207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:37507 (local address 127.3.204.67:46093)
0811 12:34:17.433388 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:17.433492 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260811 12:34:17.433841 17667 consensus_peers.cc:487] T c5ecec4212c94c869f435df7848a3f01 P 2b89b55c1c404c7db3e7776c5cc7f266 -> Peer 72d6e43777814bc39a42b13d33e58c5a (127.3.204.65:37507): Couldn't send request to peer 72d6e43777814bc39a42b13d33e58c5a. Status: Network error: Client connection negotiation failed: client connection to 127.3.204.65:37507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 12:34:17.433992 17861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:17.580981 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 17640
I20260811 12:34:17.581099 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:17.580508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:38391 (local address 127.3.204.67:42151)
0811 12:34:17.580813 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:17.580921 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.66:38391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260811 12:34:17.615401 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 16981
I20260811 12:34:17.616649 17106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:17.732952 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 16981
2026-08-11T12:34:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7395 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:34:17 dist-test-slave-lgr5 krb5kdc[18149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:34:17 dist-test-slave-lgr5 krb5kdc[18149](info): set up 2 sockets
Aug 11 12:34:17 dist-test-slave-lgr5 krb5kdc[18149](info): commencing operation
krb5kdc: starting...
W20260811 12:34:18.995476 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:34:19 dist-test-slave-lgr5 krb5kdc[18149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:34:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:34:19.263921 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:40689
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:40689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:19.599965 18172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:19.600311 18172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:19.600458 18172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:19.608237 18172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:19.608381 18172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:19.608464 18172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:19.608544 18172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:19.608606 18172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:19.618798 18172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:40689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:40689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:19.620170 18172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:19.621487 18172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:19.633181 18179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:19.633216 18180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:19.633419 18183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:19.633852 18172 server_base.cc:1034] running on GCE node
I20260811 12:34:19.634629 18172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:19.636039 18172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:19.637262 18172 hybrid_clock.cc:648] HybridClock initialized: now 1786451659637199 us; error 52 us; skew 500 ppm
I20260811 12:34:19.637604 18172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-11T12:34:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1963 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:34:19 dist-test-slave-lgr5 krb5kdc[18199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:34:19 dist-test-slave-lgr5 krb5kdc[18199](info): set up 2 sockets
Aug 11 12:34:19 dist-test-slave-lgr5 krb5kdc[18199](info): commencing operation
krb5kdc: starting...
W20260811 12:34:20.961678 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:34:21 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:34:21.243489 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:42581
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:42581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:21.576927 18219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:21.577250 18219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:21.577389 18219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:21.584620 18219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:21.584762 18219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:21.584842 18219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:21.584918 18219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:21.584981 18219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:21.594969 18219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:42581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:21.596359 18219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:21.597692 18219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:21.609472 18227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:21.609517 18228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:21.609444 18232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:21.610312 18219 server_base.cc:1034] running on GCE node
I20260811 12:34:21.611140 18219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:21.612697 18219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:21.613911 18219 hybrid_clock.cc:648] HybridClock initialized: now 1786451661613850 us; error 52 us; skew 500 ppm
I20260811 12:34:21.614261 18219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:21 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:21.617564 18219 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:34:21.619141 18219 webserver.cc:460] Webserver started at http://127.3.204.126:41429/ using document root <none> and password file <none>
I20260811 12:34:21.619902 18219 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:21.620031 18219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:21.620395 18219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:21.623624 18219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "9b6bb0454b2147568fa62952d08a050e"
format_stamp: "Formatted at 2026-08-11 12:34:21 on dist-test-slave-lgr5"
server_key: "2caffe8e5e36fb549a6e4fb9d61926c9"
server_key_iv: "edd5c81470af9be09378445c3e565e35"
server_key_version: "encryptionkey@0"
I20260811 12:34:21.624498 18219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "9b6bb0454b2147568fa62952d08a050e"
format_stamp: "Formatted at 2026-08-11 12:34:21 on dist-test-slave-lgr5"
server_key: "2caffe8e5e36fb549a6e4fb9d61926c9"
server_key_iv: "edd5c81470af9be09378445c3e565e35"
server_key_version: "encryptionkey@0"
I20260811 12:34:21.629973 18219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 12:34:21.633575 18242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:21.634914 18219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:34:21.635087 18219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "9b6bb0454b2147568fa62952d08a050e"
format_stamp: "Formatted at 2026-08-11 12:34:21 on dist-test-slave-lgr5"
server_key: "2caffe8e5e36fb549a6e4fb9d61926c9"
server_key_iv: "edd5c81470af9be09378445c3e565e35"
server_key_version: "encryptionkey@0"
I20260811 12:34:21.635277 18219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:21.653398 18219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:21.657512 18219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:21.657781 18219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:21.675717 18219 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:42581
I20260811 12:34:21.675724 18325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:42581 every 8 connection(s)
I20260811 12:34:21.677205 18219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:34:21.678581 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 18219
I20260811 12:34:21.678778 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:34:21.679414 3889 external_mini_cluster.cc:1428] Setting key 0685d4a4741cd17eb0446593fc330ce3
I20260811 12:34:21.681648 18326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 12:34:21 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:34:21.694906 18326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Bootstrap starting.
I20260811 12:34:21.699255 18326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:21.700518 18326 log.cc:824] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:21.704015 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:21.681401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.3.204.126:42581)
0811 12:34:21.682181 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:21.682193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:21.682235 (+ 42us) server_negotiation.cc:407] Connection header received
0811 12:34:21.683917 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:21.683939 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:21.684395 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:21.684739 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:21.686598 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:21.688001 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:21.689743 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:21.690475 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:21.690635 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:21.695319 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:21.695331 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:21.695352 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:21.695402 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:21.699255 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:21.700034 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:21.700044 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:21.700052 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:21.700320 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:21.700686 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:21.700691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:21.700694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:21.701430 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:21.701655 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:21.702211 (+ 556us) server_negotiation.cc:299] Negotiation successful
0811 12:34:21.702653 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":345,"threads_started":1}
I20260811 12:34:21.704154 18326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: No bootstrap required, opened a new log
I20260811 12:34:21.712071 18326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } }
I20260811 12:34:21.712572 18326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:21.712692 18326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6bb0454b2147568fa62952d08a050e, State: Initialized, Role: FOLLOWER
I20260811 12:34:21.713328 18326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } }
I20260811 12:34:21.713562 18326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:34:21.713665 18326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:34:21.713795 18326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:21.716261 18326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } }
I20260811 12:34:21.716751 18326 leader_election.cc:302] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b6bb0454b2147568fa62952d08a050e; no voters:
I20260811 12:34:21.717314 18326 leader_election.cc:288] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:34:21.717454 18334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:21.718099 18334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 1 LEADER]: Becoming Leader. State: Replica: 9b6bb0454b2147568fa62952d08a050e, State: Running, Role: LEADER
I20260811 12:34:21.718653 18334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } }
I20260811 12:34:21.719338 18326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:21.721870 18336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b6bb0454b2147568fa62952d08a050e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } } }
I20260811 12:34:21.722402 18336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:21.721843 18338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b6bb0454b2147568fa62952d08a050e. Latest consensus state: current_term: 1 leader_uuid: "9b6bb0454b2147568fa62952d08a050e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } } }
I20260811 12:34:21.723018 18338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:21.724465 18351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:34:21.730042 18351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:34:21.740831 18351 catalog_manager.cc:1347] Generated new cluster ID: 2a579175f2bf464b8ce6fd179e8daaf3
I20260811 12:34:21.740968 18351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:34:21.768532 18351 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:34:21.769979 18351 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:34:21.781277 18351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Generated new TSK 0
I20260811 12:34:21.782166 18351 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:34:21.848609 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42581
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:22.223208 18374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:22.223539 18374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:22.223695 18374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:22.231194 18374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:22.231319 18374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:22.231441 18374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:34:22.242250 18374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:22.243691 18374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:22.245097 18374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:22.258541 18381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:22.258563 18382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:22.258599 18384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:22.259716 18374 server_base.cc:1034] running on GCE node
I20260811 12:34:22.260424 18374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:22.261416 18374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:22.262647 18374 hybrid_clock.cc:648] HybridClock initialized: now 1786451662262584 us; error 51 us; skew 500 ppm
I20260811 12:34:22.262988 18374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:22 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:22.266361 18374 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:34:22.268070 18374 webserver.cc:460] Webserver started at http://127.3.204.65:42759/ using document root <none> and password file <none>
I20260811 12:34:22.268887 18374 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:22.269017 18374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:22.269341 18374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:22.272595 18374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "8df4317257524d1ebaccfe740c73e141"
format_stamp: "Formatted at 2026-08-11 12:34:22 on dist-test-slave-lgr5"
server_key: "ce25cedf0f0bf0117de901916ea77d70"
server_key_iv: "95e289c0381d3adf048bc8414ab4714f"
server_key_version: "encryptionkey@0"
I20260811 12:34:22.273447 18374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "8df4317257524d1ebaccfe740c73e141"
format_stamp: "Formatted at 2026-08-11 12:34:22 on dist-test-slave-lgr5"
server_key: "ce25cedf0f0bf0117de901916ea77d70"
server_key_iv: "95e289c0381d3adf048bc8414ab4714f"
server_key_version: "encryptionkey@0"
I20260811 12:34:22.279160 18374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260811 12:34:22.282966 18397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:22.284515 18374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:34:22.284686 18374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "8df4317257524d1ebaccfe740c73e141"
format_stamp: "Formatted at 2026-08-11 12:34:22 on dist-test-slave-lgr5"
server_key: "ce25cedf0f0bf0117de901916ea77d70"
server_key_iv: "95e289c0381d3adf048bc8414ab4714f"
server_key_version: "encryptionkey@0"
I20260811 12:34:22.284888 18374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:22.298507 18374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:22.302991 18374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:22.303268 18374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:22.304301 18374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:22.305706 18374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:22.305781 18374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:22.305902 18374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:22.305958 18374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:22.336396 18374 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:35531
I20260811 12:34:22.336444 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:35531 every 8 connection(s)
I20260811 12:34:22.337731 18374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:34:22.345584 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 18374
I20260811 12:34:22.345794 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:34:22.346413 3889 external_mini_cluster.cc:1428] Setting key e40fe4f52521da3b57c32bbb448d575a
Aug 11 12:34:22 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:34:22.359016 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:22.340327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:48853 (local address 127.3.204.126:42581)
0811 12:34:22.340769 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:22.340774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:22.341917 (+ 1143us) server_negotiation.cc:407] Connection header received
0811 12:34:22.343705 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:22.343710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:22.343781 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:22.343881 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:22.345619 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:22.346395 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:22.348135 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:22.348736 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:22.348865 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:22.352502 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:22.352509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:22.352513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:22.352555 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:22.355729 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:22.356639 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:22.356645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:22.356648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:22.356795 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:22.357224 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:22.357228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:22.357231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:22.357579 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:22.357758 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:22.358484 (+ 726us) server_negotiation.cc:299] Negotiation successful
0811 12:34:22.358712 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":233,"threads_started":1}
I20260811 12:34:22.360261 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:22.340965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:48853)
0811 12:34:22.341724 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:22.341760 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:22.343456 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:22.344104 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:22.344112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:22.344666 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:22.345412 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:22.345440 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:22.346573 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:22.346579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:22.347968 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:22.347994 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:22.348972 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:22.348987 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:22.349192 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:22.349887 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:22.349924 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:22.352272 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:22.355923 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:22.355931 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:22.355946 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:22.356507 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:22.356902 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:22.356905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:22.356908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:22.357117 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:22.357717 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:22.357723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:22.358094 (+ 371us) client_negotiation.cc:769] Sending connection context
0811 12:34:22.358334 (+ 240us) client_negotiation.cc:240] Negotiation successful
0811 12:34:22.358797 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":297,"threads_started":1}
I20260811 12:34:22.361945 18577 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42581
I20260811 12:34:22.362269 18577 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:22.363008 18577 heartbeater.cc:511] Master 127.3.204.126:42581 requested a full tablet report, sending...
I20260811 12:34:22.365469 18275 ts_manager.cc:194] Registered new tserver with Master: 8df4317257524d1ebaccfe740c73e141 (127.3.204.65:35531)
I20260811 12:34:22.367450 18275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:48853
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:34:22.407692 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42581
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:22.745250 18586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:22.745573 18586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:22.745673 18586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:22.753118 18586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:22.753238 18586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:22.753357 18586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:34:22.763670 18586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:22.765115 18586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:22.766479 18586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:22.779934 18600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:22.779961 18593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:22.780208 18592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:22.780789 18586 server_base.cc:1034] running on GCE node
I20260811 12:34:22.781422 18586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:22.782480 18586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:22.783725 18586 hybrid_clock.cc:648] HybridClock initialized: now 1786451662783650 us; error 63 us; skew 500 ppm
I20260811 12:34:22.784127 18586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:22 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:22.787555 18586 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:34:22.789353 18586 webserver.cc:460] Webserver started at http://127.3.204.66:34991/ using document root <none> and password file <none>
I20260811 12:34:22.790124 18586 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:22.790263 18586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:22.790608 18586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:22.793812 18586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "ff672d16ab6f4cf2b894455f89e33b1b"
format_stamp: "Formatted at 2026-08-11 12:34:22 on dist-test-slave-lgr5"
server_key: "237dc652a815bf711ad5d3e64e474169"
server_key_iv: "ffc4d0637d27a4288854b31492baf5c0"
server_key_version: "encryptionkey@0"
I20260811 12:34:22.794646 18586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "ff672d16ab6f4cf2b894455f89e33b1b"
format_stamp: "Formatted at 2026-08-11 12:34:22 on dist-test-slave-lgr5"
server_key: "237dc652a815bf711ad5d3e64e474169"
server_key_iv: "ffc4d0637d27a4288854b31492baf5c0"
server_key_version: "encryptionkey@0"
I20260811 12:34:22.800392 18586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 12:34:22.803953 18611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:22.805411 18586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:34:22.805590 18586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "ff672d16ab6f4cf2b894455f89e33b1b"
format_stamp: "Formatted at 2026-08-11 12:34:22 on dist-test-slave-lgr5"
server_key: "237dc652a815bf711ad5d3e64e474169"
server_key_iv: "ffc4d0637d27a4288854b31492baf5c0"
server_key_version: "encryptionkey@0"
I20260811 12:34:22.805796 18586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:22.824199 18586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:22.828723 18586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:22.829025 18586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:22.830192 18586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:22.831631 18586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:22.831728 18586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:22.831858 18586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:22.831912 18586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:22.862891 18586 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:34561
I20260811 12:34:22.862977 18803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:34561 every 8 connection(s)
I20260811 12:34:22.864291 18586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:34:22.873549 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 18586
I20260811 12:34:22.873729 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:34:22.874215 3889 external_mini_cluster.cc:1428] Setting key 0957ec78823f955b30fff9cc646d6b43
Aug 11 12:34:22 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:34:22.885454 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:22.866852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:40111 (local address 127.3.204.126:42581)
0811 12:34:22.867293 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:22.867298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:22.868276 (+ 978us) server_negotiation.cc:407] Connection header received
0811 12:34:22.870191 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:22.870197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:22.870284 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:22.870387 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:22.872123 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:22.872883 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:22.874649 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:22.875268 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:22.875399 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:22.879432 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:22.879438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:22.879442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:22.879479 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:22.882029 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:22.882965 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:22.882970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:22.882972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:22.883113 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:22.883635 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:22.883639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:22.883642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:22.883947 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:22.884063 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:22.884953 (+ 890us) server_negotiation.cc:299] Negotiation successful
0811 12:34:22.885147 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":217,"threads_started":1}
I20260811 12:34:22.886754 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:22.867376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:40111)
0811 12:34:22.868112 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:22.868169 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:22.869887 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:22.870680 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:22.870689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:22.871271 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:22.871902 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:22.871922 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:22.873067 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:22.873071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:22.874492 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:22.874509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:22.875516 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:22.875524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:22.875772 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:22.876477 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:22.876502 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:22.879138 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:22.882189 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:22.882199 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:22.882219 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:22.882823 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:22.883226 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:22.883230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:22.883233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:22.883508 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:22.884073 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:22.884106 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:22.884561 (+ 455us) client_negotiation.cc:769] Sending connection context
0811 12:34:22.884886 (+ 325us) client_negotiation.cc:240] Negotiation successful
0811 12:34:22.885418 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":265,"threads_started":1}
I20260811 12:34:22.888303 18804 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42581
I20260811 12:34:22.888664 18804 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:22.889405 18804 heartbeater.cc:511] Master 127.3.204.126:42581 requested a full tablet report, sending...
I20260811 12:34:22.890897 18275 ts_manager.cc:194] Registered new tserver with Master: ff672d16ab6f4cf2b894455f89e33b1b (127.3.204.66:34561)
I20260811 12:34:22.891701 18275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:40111
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:34:22.932397 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42581
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:23.273264 18815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:23.273662 18815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:23.273824 18815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:23.281780 18815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:23.281931 18815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:23.282097 18815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:34:23.292550 18815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:23.294102 18815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:23.295590 18815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:23.309813 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:23.310286 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:23.310536 18822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:23.311612 18815 server_base.cc:1034] running on GCE node
I20260811 12:34:23.312345 18815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:23.313381 18815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:23.314627 18815 hybrid_clock.cc:648] HybridClock initialized: now 1786451663314566 us; error 47 us; skew 500 ppm
I20260811 12:34:23.315001 18815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:23 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451663, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:23.318277 18815 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:34:23.319914 18815 webserver.cc:460] Webserver started at http://127.3.204.67:45601/ using document root <none> and password file <none>
I20260811 12:34:23.320709 18815 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:23.320840 18815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:23.321192 18815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:23.324311 18815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "9c7378d02f7e43ad98317ac2369d0153"
format_stamp: "Formatted at 2026-08-11 12:34:23 on dist-test-slave-lgr5"
server_key: "860d5c03471f37d8dedc92c4c2b3f5df"
server_key_iv: "46fcf375e53f445c0cb6a22dfde7638f"
server_key_version: "encryptionkey@0"
I20260811 12:34:23.325111 18815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "9c7378d02f7e43ad98317ac2369d0153"
format_stamp: "Formatted at 2026-08-11 12:34:23 on dist-test-slave-lgr5"
server_key: "860d5c03471f37d8dedc92c4c2b3f5df"
server_key_iv: "46fcf375e53f445c0cb6a22dfde7638f"
server_key_version: "encryptionkey@0"
I20260811 12:34:23.330508 18815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:34:23.334295 18837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:23.335723 18815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:34:23.335891 18815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "9c7378d02f7e43ad98317ac2369d0153"
format_stamp: "Formatted at 2026-08-11 12:34:23 on dist-test-slave-lgr5"
server_key: "860d5c03471f37d8dedc92c4c2b3f5df"
server_key_iv: "46fcf375e53f445c0cb6a22dfde7638f"
server_key_version: "encryptionkey@0"
I20260811 12:34:23.336135 18815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:23.360663 18815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:23.364895 18815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:23.365197 18815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:23.366312 18815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:23.367733 18815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:23.367817 18815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:23.367944 18815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:23.368008 18815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:23.370822 18577 heartbeater.cc:503] Master 127.3.204.126:42581 was elected leader, sending a full tablet report...
I20260811 12:34:23.399572 18815 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:42691
I20260811 12:34:23.399652 19041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:42691 every 8 connection(s)
I20260811 12:34:23.400942 18815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:34:23.409569 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 18815
I20260811 12:34:23.409731 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:34:23.410166 3889 external_mini_cluster.cc:1428] Setting key ac2776296d351df2f4f6b8eee899dff5
Aug 11 12:34:23 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451663, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:34:23.421078 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:23.403488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:39283 (local address 127.3.204.126:42581)
0811 12:34:23.404008 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:23.404015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:23.404899 (+ 884us) server_negotiation.cc:407] Connection header received
0811 12:34:23.406698 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:23.406702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:23.406786 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:23.406884 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:23.408438 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.409284 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.411095 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.411623 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.411722 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:23.415468 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:23.415474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:23.415477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:23.415512 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:23.417937 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:23.418641 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:23.418649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:23.418652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:23.418768 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:23.419169 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:23.419174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:23.419176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:23.419607 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:23.419750 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:23.420569 (+ 819us) server_negotiation.cc:299] Negotiation successful
0811 12:34:23.420775 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":223,"threads_started":1}
I20260811 12:34:23.422049 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:23.403985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:39283)
0811 12:34:23.404734 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:23.404780 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:23.406441 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:23.407095 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:23.407102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:23.407635 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:23.408278 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:23.408298 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.409435 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.409439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:23.410972 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:23.410998 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.411861 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.411870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:23.412175 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:23.412909 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:23.412941 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:23.415253 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:23.418070 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:23.418078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:23.418092 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:23.418542 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:23.418842 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:23.418845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:23.418847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:23.419033 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:23.419724 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:23.419730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:23.420143 (+ 413us) client_negotiation.cc:769] Sending connection context
0811 12:34:23.420393 (+ 250us) client_negotiation.cc:240] Negotiation successful
0811 12:34:23.420882 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":262,"threads_started":1}
I20260811 12:34:23.423450 19045 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42581
I20260811 12:34:23.423789 19045 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:23.424546 19045 heartbeater.cc:511] Master 127.3.204.126:42581 requested a full tablet report, sending...
I20260811 12:34:23.425933 18275 ts_manager.cc:194] Registered new tserver with Master: 9c7378d02f7e43ad98317ac2369d0153 (127.3.204.67:42691)
I20260811 12:34:23.426741 18275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:39283
I20260811 12:34:23.437855 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab.
Entry for principal oryx/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab.
WARNING: no policy specified for oryx/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab.
Entry for principal oryx/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab.
WARNING: no policy specified for oryx/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab.
Entry for principal oryx/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab.
I20260811 12:34:23.602836 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 18374
Aug 11 12:34:23 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:34:23.620108 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:23.604128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.3.204.65:35531)
0811 12:34:23.605070 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:23.605077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:23.605104 (+ 27us) server_negotiation.cc:407] Connection header received
0811 12:34:23.605166 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:23.605173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:23.605384 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:23.605539 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:23.607076 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.608422 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.609780 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.610460 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.610568 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:23.614652 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:23.614665 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:23.614680 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:23.614721 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:23.617554 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:23.618205 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:23.618212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:23.618214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:23.618341 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:23.618639 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:23.618642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:23.618644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:23.619013 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:23.619255 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:23.619508 (+ 253us) server_negotiation.cc:299] Negotiation successful
0811 12:34:23.619720 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":620,"threads_started":1}
I20260811 12:34:23.621474 18567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:23.703903 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 18374
I20260811 12:34:23.729862 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 18586
Aug 11 12:34:23 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:34:23.746318 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:23.730856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.3.204.66:34561)
0811 12:34:23.731452 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:23.731462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:23.731498 (+ 36us) server_negotiation.cc:407] Connection header received
0811 12:34:23.731572 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:23.731579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:23.731792 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:23.731939 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:23.733211 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.734220 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.735458 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.736032 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.736333 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:23.740560 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:23.740575 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:23.740592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:23.740636 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:23.743552 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:23.744366 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:23.744393 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:23.744395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:23.744524 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:23.744866 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:23.744869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:23.744871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:23.745216 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:23.745478 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:23.745742 (+ 264us) server_negotiation.cc:299] Negotiation successful
0811 12:34:23.745952 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":263,"threads_started":1}
I20260811 12:34:23.747622 18795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:23.829493 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 18586
I20260811 12:34:23.852371 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 18815
Aug 11 12:34:23 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:34:23.868897 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:23.853425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.3.204.67:42691)
0811 12:34:23.853934 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:23.853942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:23.853986 (+ 44us) server_negotiation.cc:407] Connection header received
0811 12:34:23.854049 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:23.854055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:23.854274 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:23.854396 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:23.855787 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.856881 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.858199 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:23.858753 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:23.858865 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:23.863395 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:23.863409 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:23.863426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:23.863475 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:23.866367 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:23.867081 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:23.867088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:23.867091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:23.867222 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:23.867506 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:23.867509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:23.867511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:23.867866 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:23.868135 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:23.868378 (+ 243us) server_negotiation.cc:299] Negotiation successful
0811 12:34:23.868594 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":217,"threads_started":1}
I20260811 12:34:23.870213 19033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:23.953558 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 18815
I20260811 12:34:23.979915 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 18219
I20260811 12:34:23.981447 18321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:24.087730 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 18219
I20260811 12:34:24.114883 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:42581
--webserver_interface=127.3.204.126
--webserver_port=41429
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:42581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:24.456027 19080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:24.456413 19080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:24.456542 19080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:24.463784 19080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:24.463900 19080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:24.463956 19080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:24.464042 19080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:24.464116 19080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:24.474399 19080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:42581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:24.475822 19080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:24.477185 19080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:24.488926 19091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:24.488833 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:24.488865 19092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:24.490211 19080 server_base.cc:1034] running on GCE node
I20260811 12:34:24.491021 19080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:24.492563 19080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:24.493755 19080 hybrid_clock.cc:648] HybridClock initialized: now 1786451664493696 us; error 51 us; skew 500 ppm
I20260811 12:34:24.494105 19080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:24 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:24.497526 19080 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:34:24.498999 19080 webserver.cc:460] Webserver started at http://127.3.204.126:41429/ using document root <none> and password file <none>
I20260811 12:34:24.499647 19080 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:24.499758 19080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:24.505097 19080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:34:24.507992 19106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:24.509366 19080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:34:24.509528 19080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "9b6bb0454b2147568fa62952d08a050e"
format_stamp: "Formatted at 2026-08-11 12:34:21 on dist-test-slave-lgr5"
server_key: "2caffe8e5e36fb549a6e4fb9d61926c9"
server_key_iv: "edd5c81470af9be09378445c3e565e35"
server_key_version: "encryptionkey@0"
I20260811 12:34:24.510462 19080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:24.525935 19080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:24.530285 19080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:24.530527 19080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:24.547469 19080 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:42581
I20260811 12:34:24.547487 19203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:42581 every 8 connection(s)
I20260811 12:34:24.549057 19080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:34:24.550273 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 19080
I20260811 12:34:24.551052 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:35531
--local_ip_for_outbound_sockets=127.3.204.65
--tserver_master_addrs=127.3.204.126:42581
--webserver_port=42759
--webserver_interface=127.3.204.65
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.204.65
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:34:24.556452 19205 sys_catalog.cc:263] Verifying existing consensus state
I20260811 12:34:24.559918 19205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Bootstrap starting.
I20260811 12:34:24.585757 19205 log.cc:824] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:24.597443 19205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:34:24.598096 19205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Bootstrap complete.
I20260811 12:34:24.611248 19205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } }
I20260811 12:34:24.612566 19205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6bb0454b2147568fa62952d08a050e, State: Initialized, Role: FOLLOWER
I20260811 12:34:24.613376 19205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } }
I20260811 12:34:24.613708 19205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:34:24.613840 19205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:34:24.614051 19205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 1 FOLLOWER]: Advancing to term 2
I20260811 12:34:24.620035 19205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } }
I20260811 12:34:24.620731 19205 leader_election.cc:302] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b6bb0454b2147568fa62952d08a050e; no voters:
I20260811 12:34:24.621414 19205 leader_election.cc:288] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 12:34:24.621543 19210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 2 FOLLOWER]: Leader election won for term 2
I20260811 12:34:24.622742 19210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [term 2 LEADER]: Becoming Leader. State: Replica: 9b6bb0454b2147568fa62952d08a050e, State: Running, Role: LEADER
I20260811 12:34:24.623322 19210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } }
I20260811 12:34:24.623899 19205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:24.625618 19212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b6bb0454b2147568fa62952d08a050e. Latest consensus state: current_term: 2 leader_uuid: "9b6bb0454b2147568fa62952d08a050e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } } }
I20260811 12:34:24.625980 19212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:24.626515 19211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b6bb0454b2147568fa62952d08a050e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6bb0454b2147568fa62952d08a050e" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42581 } } }
I20260811 12:34:24.626808 19211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:24.637364 19215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:34:24.643075 19215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:34:24.644548 19215 catalog_manager.cc:1259] Loaded cluster ID: 2a579175f2bf464b8ce6fd179e8daaf3
I20260811 12:34:24.644712 19215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:34:24.648667 19215 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:34:24.650216 19215 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b6bb0454b2147568fa62952d08a050e: Loaded TSK: 0
I20260811 12:34:24.651906 19215 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 12:34:24.929042 19208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:24.929354 19208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:24.929488 19208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:24.937047 19208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:24.937163 19208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:24.937307 19208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:34:24.948513 19208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.65.keytab
--principal=oryx/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:35531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=42759
--tserver_master_addrs=127.3.204.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:24.950074 19208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:24.951536 19208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:24.965709 19243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:24.965875 19242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:24.965931 19245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:24.966338 19208 server_base.cc:1034] running on GCE node
I20260811 12:34:24.967137 19208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:24.968202 19208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:24.969421 19208 hybrid_clock.cc:648] HybridClock initialized: now 1786451664969375 us; error 42 us; skew 500 ppm
I20260811 12:34:24.969731 19208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:24 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451664, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:24.972904 19208 init.cc:377] Logged in from keytab as oryx/127.3.204.65@KRBTEST.COM (short username oryx)
I20260811 12:34:24.974470 19208 webserver.cc:460] Webserver started at http://127.3.204.65:42759/ using document root <none> and password file <none>
I20260811 12:34:24.975183 19208 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:24.975312 19208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:24.980997 19208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:34:24.983692 19255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:24.985109 19208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:34:24.985258 19208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "8df4317257524d1ebaccfe740c73e141"
format_stamp: "Formatted at 2026-08-11 12:34:22 on dist-test-slave-lgr5"
server_key: "ce25cedf0f0bf0117de901916ea77d70"
server_key_iv: "95e289c0381d3adf048bc8414ab4714f"
server_key_version: "encryptionkey@0"
I20260811 12:34:24.986219 19208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:25.018683 19208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:25.023094 19208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:25.023360 19208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:25.024453 19208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:25.025831 19208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:25.025918 19208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:25.026017 19208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:25.026093 19208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:25.061266 19208 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:35531
I20260811 12:34:25.061391 19462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:35531 every 8 connection(s)
I20260811 12:34:25.062960 19208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:34:25.064008 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 19208
I20260811 12:34:25.064751 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:34561
--local_ip_for_outbound_sockets=127.3.204.66
--tserver_master_addrs=127.3.204.126:42581
--webserver_port=34991
--webserver_interface=127.3.204.66
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.204.66
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:34:25 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:25 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 12:34:25.408643 19470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:25.408953 19470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:25.409101 19470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:25.416790 19470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:25.416916 19470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:25.417088 19470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:34:25.427310 19470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.66.keytab
--principal=oryx/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:34561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=34991
--tserver_master_addrs=127.3.204.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:25.429173 19470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:25.430522 19470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:25.447280 19483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:25.447925 19487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:25.447237 19482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:25.453259 19470 server_base.cc:1034] running on GCE node
I20260811 12:34:25.454216 19470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:25.455654 19470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:25.456913 19470 hybrid_clock.cc:648] HybridClock initialized: now 1786451665456844 us; error 51 us; skew 500 ppm
I20260811 12:34:25.457365 19470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:25 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451665, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:25.461793 19470 init.cc:377] Logged in from keytab as oryx/127.3.204.66@KRBTEST.COM (short username oryx)
I20260811 12:34:25.463514 19470 webserver.cc:460] Webserver started at http://127.3.204.66:34991/ using document root <none> and password file <none>
I20260811 12:34:25.464359 19470 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:25.464500 19470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:25.470682 19470 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 12:34:25.474012 19499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:25.475659 19470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:34:25.475822 19470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "ff672d16ab6f4cf2b894455f89e33b1b"
format_stamp: "Formatted at 2026-08-11 12:34:22 on dist-test-slave-lgr5"
server_key: "237dc652a815bf711ad5d3e64e474169"
server_key_iv: "ffc4d0637d27a4288854b31492baf5c0"
server_key_version: "encryptionkey@0"
I20260811 12:34:25.476879 19470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:25.533279 19470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:25.537875 19470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:25.538194 19470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:25.539381 19470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:25.540827 19470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:25.540933 19470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:25.541038 19470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:25.541111 19470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:25.582196 19470 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:34561
I20260811 12:34:25.582243 19681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:34561 every 8 connection(s)
I20260811 12:34:25.583768 19470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:34:25.593047 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 19470
I20260811 12:34:25.593901 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:42691
--local_ip_for_outbound_sockets=127.3.204.67
--tserver_master_addrs=127.3.204.126:42581
--webserver_port=45601
--webserver_interface=127.3.204.67
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.204.67
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:34:25 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:25 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 12:34:25.943571 19690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:25.943907 19690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:25.944100 19690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:25.953414 19690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:25.953548 19690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:25.953706 19690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:34:25.963857 19690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/krb5kdc/oryx_127.3.204.67.keytab
--principal=oryx/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:42691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=45601
--tserver_master_addrs=127.3.204.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:25.965838 19690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:25.967149 19690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:25.993074 19700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:25.996925 19699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:26.009958 19702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:26.010654 19690 server_base.cc:1034] running on GCE node
I20260811 12:34:26.011678 19690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:26.013305 19690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:26.014744 19690 hybrid_clock.cc:648] HybridClock initialized: now 1786451666014680 us; error 100 us; skew 500 ppm
I20260811 12:34:26.015257 19690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451666, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:26.023821 19690 init.cc:377] Logged in from keytab as oryx/127.3.204.67@KRBTEST.COM (short username oryx)
I20260811 12:34:26.026451 19690 webserver.cc:460] Webserver started at http://127.3.204.67:45601/ using document root <none> and password file <none>
I20260811 12:34:26.027650 19690 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:26.027859 19690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:26.037273 19690 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 12:34:26.041826 19711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:26.044194 19690 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.002s
I20260811 12:34:26.044445 19690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "9c7378d02f7e43ad98317ac2369d0153"
format_stamp: "Formatted at 2026-08-11 12:34:23 on dist-test-slave-lgr5"
server_key: "860d5c03471f37d8dedc92c4c2b3f5df"
server_key_iv: "46fcf375e53f445c0cb6a22dfde7638f"
server_key_version: "encryptionkey@0"
I20260811 12:34:26.046159 19690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:26.067628 19690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:26.074359 19690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:26.074795 19690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:26.076392 19690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:26.078536 19690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:26.078720 19690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260811 12:34:26.078882 19690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:26.079005 19690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:26.154202 19690 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:42691
I20260811 12:34:26.154249 19896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:42691 every 8 connection(s)
I20260811 12:34:26.155892 19690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:34:26.164896 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 19690
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:26.202562 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:26.167076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.3.204.126:42581)
0811 12:34:26.184549 (+ 17473us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:26.184554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:26.184574 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:34:26.184637 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:26.184642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:26.184722 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:26.184838 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:26.187366 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.188672 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.190868 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.191537 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.191673 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.195101 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:26.195111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:26.195132 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:26.195184 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:26.198369 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:26.199151 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:26.199160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:26.199167 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:26.199347 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:26.199725 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:26.199729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:26.199731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:26.200357 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:26.200682 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:26.201151 (+ 469us) server_negotiation.cc:299] Negotiation successful
0811 12:34:26.201672 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17317,"thread_start_us":17181,"threads_started":1}
W20260811 12:34:26.367779 19469 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.289s user 0.003s sys 0.011s
W20260811 12:34:26.369841 19467 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:25.066377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:58863 (local address 127.3.204.126:42581)
0811 12:34:25.067125 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:25.067141 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:25.067930 (+ 789us) server_negotiation.cc:407] Connection header received
0811 12:34:25.070642 (+ 2712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:25.070656 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:25.071014 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:25.071327 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:25.073268 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:25.074625 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:25.076386 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:25.077078 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:25.077265 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.369438 (+1292173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:58863: BlockingRecv error: recv got EOF from 127.3.204.65:58863 (error 108)
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":265,"threads_started":1}
I20260811 12:34:26.370786 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:25.066386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:58863)
0811 12:34:25.067714 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:25.067775 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:25.070013 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:25.071550 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:25.071561 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:25.072319 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:25.073071 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:25.073098 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:25.074826 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:25.074831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:25.076226 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:25.076239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:25.077334 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:25.077342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:25.077605 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:25.078256 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:25.078290 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:26.368975 (+1290685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":804,"threads_started":1}
W20260811 12:34:26.372052 19464 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:42581 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.204.126:42581: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 12:34:26.387300 19467 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:26.373853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:51963 (local address 127.3.204.126:42581)
0811 12:34:26.374091 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:26.374098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:26.374120 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:34:26.374191 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:26.374197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:26.374288 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:26.374394 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:26.375778 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.377107 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.379286 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.380025 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.380213 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.386962 (+ 6749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:51963: BlockingRecv error: recv got EOF from 127.3.204.65:51963 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 12:34:26.388245 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:26.373039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:51963)
0811 12:34:26.373286 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:26.373312 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:26.373472 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:26.374666 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:26.374672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:26.375259 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:26.375558 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.375571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.377275 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.377282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.379073 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.379093 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.380279 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.380287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.380512 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.381348 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:26.381375 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:26.387868 (+ 6493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:26.402590 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:26.389228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:40563)
0811 12:34:26.389513 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:26.389535 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:26.389657 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:26.390020 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:26.390025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:26.390559 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:26.390817 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.390830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.392505 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.392511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.394320 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.394339 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.395489 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.395496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.395744 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.396594 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:26.396621 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:26.402249 (+ 5628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159}
W20260811 12:34:26.402861 19467 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:26.389396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:40563 (local address 127.3.204.126:42581)
0811 12:34:26.389655 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:26.389663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:26.389686 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:34:26.389759 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:26.389765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:26.389852 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:26.389968 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:26.391007 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.392337 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.394501 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.395309 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.395455 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.402547 (+ 7092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:40563: BlockingRecv error: recv got EOF from 127.3.204.65:40563 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
W20260811 12:34:26.403272 19464 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 12:34:26.745712 19688 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.149s user 0.001s sys 0.012s
W20260811 12:34:26.747265 19687 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:25.586849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:41655 (local address 127.3.204.126:42581)
0811 12:34:25.587304 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:25.587308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:25.588740 (+ 1432us) server_negotiation.cc:407] Connection header received
0811 12:34:25.590717 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:25.590723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:25.590825 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:25.590927 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:25.592583 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:25.593433 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:25.595180 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:25.595595 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:25.595711 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.746858 (+1151147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:41655: BlockingRecv error: recv got EOF from 127.3.204.66:41655 (error 108)
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260811 12:34:26.747439 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:25.587699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:41655)
0811 12:34:25.588567 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:25.588616 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:25.590439 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:25.591217 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:25.591227 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:25.591755 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:25.592387 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:25.592407 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:25.593613 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:25.593617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:25.595031 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:25.595045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:25.595826 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:25.595834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:25.596074 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:25.596768 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:25.596793 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:26.746518 (+1149725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":363,"threads_started":1}
W20260811 12:34:26.748406 19682 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:42581 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.204.126:42581: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 12:34:26.759052 19687 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:26.749164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:51735 (local address 127.3.204.126:42581)
0811 12:34:26.749326 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:26.749332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:26.749487 (+ 155us) server_negotiation.cc:407] Connection header received
0811 12:34:26.749671 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:26.749675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:26.749737 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:26.749827 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:26.750604 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.751345 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.752697 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.753197 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.753311 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.758811 (+ 5500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:51735: BlockingRecv error: recv got EOF from 127.3.204.66:51735 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 12:34:26.759605 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:26.749165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:51735)
0811 12:34:26.749402 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:26.749418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:26.749549 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:26.749864 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:26.749868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:26.750269 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:26.750460 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.750469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.751492 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.751498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.752558 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.752571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.753375 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.753382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.753540 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.754159 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:26.754177 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:26.759383 (+ 5206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:26 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:26.769186 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:26.760500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:51317)
0811 12:34:26.760823 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:26.760842 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:26.760962 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:26.761269 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:26.761273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:26.761618 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:26.761781 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.761790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.762787 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.762793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.763753 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.763765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.764511 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.764518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.764704 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.765247 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:26.765265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:26.768857 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213}
W20260811 12:34:26.769299 19687 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:26.760651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:51317 (local address 127.3.204.126:42581)
0811 12:34:26.760824 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:26.760829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:26.760902 (+ 73us) server_negotiation.cc:407] Connection header received
0811 12:34:26.761067 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:26.761073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:26.761138 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:26.761218 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:26.761920 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.762642 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.763890 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.764364 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.764474 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.769039 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:51317: BlockingRecv error: recv got EOF from 127.3.204.66:51317 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260811 12:34:26.769593 19682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 12:34:27.200954 19904 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.022s user 0.015s sys 0.000s
W20260811 12:34:27.202164 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:26.159015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:38603 (local address 127.3.204.126:42581)
0811 12:34:26.159453 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:26.159460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:26.161067 (+ 1607us) server_negotiation.cc:407] Connection header received
0811 12:34:26.163147 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:26.163153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:26.163242 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:26.163380 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:26.165435 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.166662 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.175874 (+ 9212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.176679 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.176799 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.201945 (+1025146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:38603: BlockingRecv error: recv got EOF from 127.3.204.67:38603 (error 108)
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":231,"threads_started":1}
I20260811 12:34:27.202692 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:26.159959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:38603)
0811 12:34:26.160886 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:26.160928 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:26.162826 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:26.163657 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:26.163670 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:26.164301 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:26.165172 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.165201 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.173148 (+ 7947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.173155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.175606 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:26.175640 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:26.177319 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:26.177328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:26.177718 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:26.178752 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:26.178791 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:27.201726 (+1022935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":365,"threads_started":1}
W20260811 12:34:27.203652 19897 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:42581 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.204.126:42581: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Aug 11 12:34:27 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:27 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:27.212870 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:27.204278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:35273)
0811 12:34:27.204478 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:27.204497 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:27.204620 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:27.204896 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:27.204899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:27.205256 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:27.205431 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:27.205440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.206489 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.206495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:27.207820 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:27.207839 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.208612 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.208617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:27.208741 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.209278 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:27.209297 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:27.212649 (+ 3352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260811 12:34:27.213105 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:27.204460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:35273 (local address 127.3.204.126:42581)
0811 12:34:27.204643 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:27.204647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:27.204667 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:34:27.204717 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:27.204720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:27.204781 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:27.204879 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:27.205550 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.206327 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.207972 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.208447 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.208540 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.212875 (+ 4335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:35273: BlockingRecv error: recv got EOF from 127.3.204.67:35273 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 11 12:34:27 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:27 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:27.222252 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:27.213627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:49451)
0811 12:34:27.213781 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:27.213802 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:27.213914 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:27.214257 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:27.214260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:27.214602 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:27.214767 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:27.214776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.215761 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.215765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:27.216752 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:27.216765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.217519 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.217524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:27.217696 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.218308 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:27.218334 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:27.222046 (+ 3712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 12:34:27.222503 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:27.213718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:49451 (local address 127.3.204.126:42581)
0811 12:34:27.214002 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:27.214006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:27.214022 (+ 16us) server_negotiation.cc:407] Connection header received
0811 12:34:27.214068 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:27.214071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:27.214142 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:27.214221 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:27.214882 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.215648 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.216875 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.217325 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.217418 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.222248 (+ 4830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:49451: BlockingRecv error: recv got EOF from 127.3.204.67:49451 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
W20260811 12:34:27.222559 19897 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 12:34:27 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:27 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:27.412720 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:27.403908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:35493)
0811 12:34:27.404584 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:27.404599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:27.404712 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:27.405007 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:27.405010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:27.405442 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:27.405595 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:27.405603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.406535 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.406539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:27.407533 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:27.407546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.408287 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.408292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:27.408418 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.409030 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:27.409057 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:27.412466 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":439,"threads_started":1}
W20260811 12:34:27.412952 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:27.403962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:35493 (local address 127.3.204.126:42581)
0811 12:34:27.404202 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:27.404209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:27.404674 (+ 465us) server_negotiation.cc:407] Connection header received
0811 12:34:27.404817 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:27.404821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:27.404902 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:27.404989 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:27.405710 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.406410 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.407648 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.408139 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.408233 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.412667 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:35493: BlockingRecv error: recv got EOF from 127.3.204.65:35493 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 11 12:34:27 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:27 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:27.779117 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:27.770195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:46523)
0811 12:34:27.770751 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:27.770768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:27.770890 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:27.771175 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:27.771179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:27.771626 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:27.771779 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:27.771788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.772857 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.772862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:27.773832 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:27.773845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.774512 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.774517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:27.774650 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.775234 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:27.775254 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:27.778776 (+ 3522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":328,"threads_started":1}
W20260811 12:34:27.779325 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:27.770319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:46523 (local address 127.3.204.126:42581)
0811 12:34:27.770486 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:27.770492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:27.770833 (+ 341us) server_negotiation.cc:407] Connection header received
0811 12:34:27.770995 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:27.770999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:27.771062 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:27.771145 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:27.771889 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.772712 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.773943 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:27.774346 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:27.774435 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:27.779064 (+ 4629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:46523: BlockingRecv error: recv got EOF from 127.3.204.66:46523 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 11 12:34:28 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:28 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:28.232191 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:28.223172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:36539)
0811 12:34:28.223723 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:28.223740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:28.223888 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:28.224267 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:28.224271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:28.224738 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:28.224899 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:28.224909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.225940 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.225946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:28.226894 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:28.226906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.227668 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.227676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:28.227842 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:28.228436 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:28.228454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:28.231805 (+ 3351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":327,"threads_started":1}
W20260811 12:34:28.232303 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:28.223262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:36539 (local address 127.3.204.126:42581)
0811 12:34:28.223466 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:28.223472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:28.223826 (+ 354us) server_negotiation.cc:407] Connection header received
0811 12:34:28.224030 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:28.224035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:28.224133 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:28.224218 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:28.225064 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.225807 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.227034 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.227490 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.227599 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:28.232071 (+ 4472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:36539: BlockingRecv error: recv got EOF from 127.3.204.67:36539 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 11 12:34:28 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:28 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:28.422937 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:28.413749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:41717)
0811 12:34:28.414294 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:28.414311 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:28.414456 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:28.414830 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:28.414833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:28.415269 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:28.415466 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:28.415476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.416557 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.416561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:28.417498 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:28.417511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.418204 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.418210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:28.418381 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:28.419001 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:28.419021 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:28.422596 (+ 3575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":301,"threads_started":1}
W20260811 12:34:28.423177 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:28.413821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:41717 (local address 127.3.204.126:42581)
0811 12:34:28.414028 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:28.414033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:28.414421 (+ 388us) server_negotiation.cc:407] Connection header received
0811 12:34:28.414580 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:28.414586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:28.414677 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:28.414795 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:28.415622 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.416428 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.417611 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.418021 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.418130 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:28.422878 (+ 4748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:41717: BlockingRecv error: recv got EOF from 127.3.204.65:41717 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 11 12:34:28 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:28 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:28.789625 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:28.780262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:36851)
0811 12:34:28.780834 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:28.780851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:28.780992 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:28.781291 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:28.781295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:28.781726 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:28.781958 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:28.781969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.782949 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.782953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:28.783919 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:28.783931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.784736 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.784742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:28.784910 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:28.785784 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:28.785802 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:28.789254 (+ 3452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":343,"threads_started":1}
W20260811 12:34:28.789757 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:28.780394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:36851 (local address 127.3.204.126:42581)
0811 12:34:28.780653 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:28.780659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:28.780930 (+ 271us) server_negotiation.cc:407] Connection header received
0811 12:34:28.781094 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:28.781097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:28.781164 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:28.781248 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:28.782096 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.782834 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.784105 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:28.784566 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:28.784681 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:28.789518 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:36851: BlockingRecv error: recv got EOF from 127.3.204.66:36851 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 11 12:34:29 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:29 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:29.242445 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:29.233279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:44755)
0811 12:34:29.233805 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:29.233821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:29.233942 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:29.234266 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:29.234269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:29.234673 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:29.234876 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:29.234886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.235904 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.235908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:29.236899 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:29.236912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.237711 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.237718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:29.237909 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:29.238645 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:29.238666 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:29.242104 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":290,"threads_started":1}
W20260811 12:34:29.242583 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:29.233443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44755 (local address 127.3.204.126:42581)
0811 12:34:29.233613 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:29.233618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:29.233894 (+ 276us) server_negotiation.cc:407] Connection header received
0811 12:34:29.234064 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:29.234068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:29.234147 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:29.234231 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:29.235036 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.235790 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.237040 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.237539 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.237652 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:29.242322 (+ 4670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:44755: BlockingRecv error: recv got EOF from 127.3.204.67:44755 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 11 12:34:29 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:29 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:29.433233 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:29.424048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:37079)
0811 12:34:29.424652 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:29.424672 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:29.424834 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:29.425324 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:29.425327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:29.425832 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:29.426010 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:29.426021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.427080 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.427084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:29.428018 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:29.428030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.428766 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.428771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:29.428928 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:29.429555 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:29.429573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:29.432962 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":276,"threads_started":1}
W20260811 12:34:29.433468 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:29.424143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:37079 (local address 127.3.204.126:42581)
0811 12:34:29.424321 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:29.424328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:29.424800 (+ 472us) server_negotiation.cc:407] Connection header received
0811 12:34:29.425054 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:29.425080 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:29.425169 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:29.425297 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:29.426176 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.426957 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.428180 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.428629 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.428722 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:29.433222 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:37079: BlockingRecv error: recv got EOF from 127.3.204.65:37079 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 11 12:34:29 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:29 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:29.802654 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:29.790596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:46535)
0811 12:34:29.791286 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:29.791303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:29.791471 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:29.791806 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:29.791810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:29.792291 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:29.792544 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:29.792559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.793597 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.793601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:29.794505 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:29.794516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.795164 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.795167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:29.795288 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:29.799031 (+ 3743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:29.799052 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:29.802385 (+ 3333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":428,"threads_started":1}
W20260811 12:34:29.802852 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:29.790744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:46535 (local address 127.3.204.126:42581)
0811 12:34:29.790949 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:29.790956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:29.791392 (+ 436us) server_negotiation.cc:407] Connection header received
0811 12:34:29.791609 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:29.791613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:29.791678 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:29.791770 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:29.792714 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.793483 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.794613 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:29.795055 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:29.795145 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:29.802623 (+ 7478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:46535: BlockingRecv error: recv got EOF from 127.3.204.66:46535 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 11 12:34:30 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:30 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:30.252254 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:30.243490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:44099)
0811 12:34:30.244070 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:30.244106 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:30.244226 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:30.244500 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:30.244503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:30.244881 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:30.245060 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:30.245070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.246105 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.246109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:30.247031 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:30.247044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.247777 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.247781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:30.247933 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:30.248581 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:30.248598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:30.251893 (+ 3295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":312,"threads_started":1}
W20260811 12:34:30.252415 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:30.243598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44099 (local address 127.3.204.126:42581)
0811 12:34:30.243759 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:30.243764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:30.244175 (+ 411us) server_negotiation.cc:407] Connection header received
0811 12:34:30.244315 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:30.244319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:30.244381 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:30.244463 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:30.245186 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.245992 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.247197 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.247642 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.247775 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:30.252193 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:44099: BlockingRecv error: recv got EOF from 127.3.204.67:44099 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 12:34:30 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:30 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:30.443358 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:30.434417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:43085)
0811 12:34:30.434952 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:30.434969 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:30.435107 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:30.435434 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:30.435438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:30.435841 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:30.436023 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:30.436033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.437313 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.437319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:30.438474 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:30.438498 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.439220 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.439225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:30.439379 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:30.439948 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:30.439965 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:30.443107 (+ 3142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
W20260811 12:34:30.443588 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:30.434547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:43085 (local address 127.3.204.126:42581)
0811 12:34:30.434754 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:30.434759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:30.435057 (+ 298us) server_negotiation.cc:407] Connection header received
0811 12:34:30.435213 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:30.435217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:30.435281 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:30.435370 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:30.436222 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.437144 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.438628 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.439094 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.439194 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:30.443348 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:43085: BlockingRecv error: recv got EOF from 127.3.204.65:43085 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 11 12:34:30 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:30 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:30.812412 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:30.803769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:48353)
0811 12:34:30.804343 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:30.804377 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:30.804495 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:30.804773 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:30.804776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:30.805175 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:30.805374 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:30.805385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.806389 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.806394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:30.807402 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:30.807415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.808044 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.808048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:30.808187 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:30.808754 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:30.808772 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:30.812154 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":331,"threads_started":1}
W20260811 12:34:30.812547 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:30.803774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:48353 (local address 127.3.204.126:42581)
0811 12:34:30.803960 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:30.803964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:30.804446 (+ 482us) server_negotiation.cc:407] Connection header received
0811 12:34:30.804593 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:30.804597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:30.804660 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:30.804742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:30.805503 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.806258 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.807516 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:30.807938 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:30.808028 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:30.812339 (+ 4311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:48353: BlockingRecv error: recv got EOF from 127.3.204.66:48353 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 11 12:34:31 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:31 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:31.264920 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:31.253317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:57427)
0811 12:34:31.253827 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:31.253848 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:31.253983 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:31.254274 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:31.254277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:31.254692 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:31.254864 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:31.254874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.255904 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.255907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:31.256931 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:31.256944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.257611 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.257615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:31.257743 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:31.260976 (+ 3233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:31.260996 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:31.264613 (+ 3617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
W20260811 12:34:31.265136 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:31.253422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:57427 (local address 127.3.204.126:42581)
0811 12:34:31.253673 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:31.253680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:31.253937 (+ 257us) server_negotiation.cc:407] Connection header received
0811 12:34:31.254086 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:31.254090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:31.254155 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:31.254237 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:31.255011 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.255775 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.257057 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.257495 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.257590 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:31.264863 (+ 7273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:57427: BlockingRecv error: recv got EOF from 127.3.204.67:57427 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 11 12:34:31 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:31 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:31.453450 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:31.444418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:37571)
0811 12:34:31.445004 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:31.445020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:31.445160 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:31.445505 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:31.445510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:31.445956 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:31.446125 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:31.446135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.447282 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.447286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:31.448300 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:31.448314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.448935 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.448939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:31.449075 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:31.449652 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:31.449669 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:31.453206 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":294,"threads_started":1}
W20260811 12:34:31.453672 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:31.444582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:37571 (local address 127.3.204.126:42581)
0811 12:34:31.444791 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:31.444798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:31.445119 (+ 321us) server_negotiation.cc:407] Connection header received
0811 12:34:31.445291 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:31.445296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:31.445363 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:31.445448 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:31.446261 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.447160 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.448430 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.448819 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.448911 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:31.453452 (+ 4541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:37571: BlockingRecv error: recv got EOF from 127.3.204.65:37571 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 11 12:34:31 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:31 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:31.822799 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:31.813610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:43013)
0811 12:34:31.814162 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:31.814179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:31.814330 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:31.814670 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:31.814673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:31.815205 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:31.815384 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:31.815394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.816428 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.816432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:31.817396 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:31.817409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.818131 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.818136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:31.818288 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:31.818908 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:31.818929 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:31.822570 (+ 3641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":298,"threads_started":1}
W20260811 12:34:31.823040 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:31.813794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:43013 (local address 127.3.204.126:42581)
0811 12:34:31.814017 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:31.814023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:31.814263 (+ 240us) server_negotiation.cc:407] Connection header received
0811 12:34:31.814452 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:31.814457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:31.814527 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:31.814622 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:31.815521 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.816298 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.817509 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:31.817963 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:31.818067 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:31.822788 (+ 4721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:43013: BlockingRecv error: recv got EOF from 127.3.204.66:43013 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 11 12:34:32 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:32 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:32.275139 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:32.265926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:44443)
0811 12:34:32.266495 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:32.266512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:32.266652 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:32.267037 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:32.267043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:32.267495 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:32.267644 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:32.267654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.268723 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.268727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:32.269666 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:32.269679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.270390 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.270396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:32.270576 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:32.271171 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:32.271187 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:32.274781 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":295,"threads_started":1}
W20260811 12:34:32.275213 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:32.266081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44443 (local address 127.3.204.126:42581)
0811 12:34:32.266304 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:32.266311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:32.266594 (+ 283us) server_negotiation.cc:407] Connection header received
0811 12:34:32.266766 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:32.266772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:32.266877 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:32.266980 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:32.267789 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.268583 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.269805 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.270247 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.270360 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:32.275023 (+ 4663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:44443: BlockingRecv error: recv got EOF from 127.3.204.67:44443 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 11 12:34:32 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:32 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:32.463287 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:32.454576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:38921)
0811 12:34:32.455168 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:32.455188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:32.455339 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:32.455908 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:32.455911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:32.456375 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:32.456565 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:32.456574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.457570 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.457573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:32.458482 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:32.458494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.459119 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.459123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:32.459255 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:32.459780 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:32.459798 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:32.463057 (+ 3259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":337,"threads_started":1}
W20260811 12:34:32.463549 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:32.454676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:38921 (local address 127.3.204.126:42581)
0811 12:34:32.454855 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:32.454861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:32.455276 (+ 415us) server_negotiation.cc:407] Connection header received
0811 12:34:32.455507 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:32.455513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:32.455597 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:32.455669 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:32.456699 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.457454 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.458588 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.458996 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.459088 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:32.463281 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:38921: BlockingRecv error: recv got EOF from 127.3.204.65:38921 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 11 12:34:32 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:32 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:32.832924 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:32.823802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:43351)
0811 12:34:32.824343 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:32.824359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:32.824494 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:32.824823 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:32.824826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:32.825310 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:32.825514 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:32.825525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.826523 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.826527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:32.827468 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:32.827490 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.828218 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.828222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:32.828349 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:32.828922 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:32.828940 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:32.832592 (+ 3652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
W20260811 12:34:32.833076 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:32.823877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:43351 (local address 127.3.204.126:42581)
0811 12:34:32.824119 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:32.824127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:32.824457 (+ 330us) server_negotiation.cc:407] Connection header received
0811 12:34:32.824603 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:32.824606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:32.824707 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:32.824791 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:32.825635 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.826345 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.827607 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:32.828049 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:32.828187 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:32.832873 (+ 4686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:43351: BlockingRecv error: recv got EOF from 127.3.204.66:43351 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 11 12:34:33 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:33 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:33.285387 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:33.276201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:47149)
0811 12:34:33.276751 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:33.276771 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:33.276911 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:33.277246 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:33.277251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:33.277707 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:33.277884 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:33.277893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.278975 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.278979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:33.279970 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:33.279983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.280762 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.280768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:33.280948 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:33.281543 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:33.281560 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:33.285143 (+ 3583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":314,"threads_started":1}
W20260811 12:34:33.285655 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:33.276281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:47149 (local address 127.3.204.126:42581)
0811 12:34:33.276473 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:33.276493 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:33.276871 (+ 378us) server_negotiation.cc:407] Connection header received
0811 12:34:33.277042 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:33.277048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:33.277118 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:33.277201 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:33.278045 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.278815 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.280143 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.280599 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.280721 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:33.285381 (+ 4660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:47149: BlockingRecv error: recv got EOF from 127.3.204.67:47149 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 11 12:34:33 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:33 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:33.473886 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:33.464294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:54609)
0811 12:34:33.464944 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:33.464962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:33.465084 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:33.465447 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:33.465452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:33.465891 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:33.466083 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:33.466093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.467253 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.467258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:33.468302 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:33.468316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.469122 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.469127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:33.469251 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:33.469859 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:33.469879 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:33.473635 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":305,"threads_started":1}
W20260811 12:34:33.474184 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:33.464444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:54609 (local address 127.3.204.126:42581)
0811 12:34:33.464640 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:33.464644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:33.465031 (+ 387us) server_negotiation.cc:407] Connection header received
0811 12:34:33.465185 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:33.465189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:33.465256 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:33.465342 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:33.466270 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.467087 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.468453 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.468946 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.469062 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:33.473917 (+ 4855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:54609: BlockingRecv error: recv got EOF from 127.3.204.65:54609 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 11 12:34:33 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:33 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:33.843801 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:33.834125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:41963)
0811 12:34:33.834784 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:33.834801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:33.834925 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:33.835227 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:33.835231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:33.835669 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:33.835850 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:33.835860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.837010 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.837016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:33.838012 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:33.838028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.838772 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.838777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:33.838939 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:33.839670 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:33.839694 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:33.843478 (+ 3784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":297,"threads_started":1}
W20260811 12:34:33.844012 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:33.834259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:41963 (local address 127.3.204.126:42581)
0811 12:34:33.834476 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:33.834481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:33.834874 (+ 393us) server_negotiation.cc:407] Connection header received
0811 12:34:33.835036 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:33.835039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:33.835105 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:33.835197 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:33.836003 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.836843 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.838173 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:33.838630 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:33.838722 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:33.843739 (+ 5017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:41963: BlockingRecv error: recv got EOF from 127.3.204.66:41963 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 11 12:34:34 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:34 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:34.295382 19952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:34.286395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:42093)
0811 12:34:34.286989 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:34.287006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:34.287130 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:34.287406 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:34.287409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:34.287857 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:34.288038 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:34.288048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.289014 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.289017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:34.290026 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:34.290040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.290772 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.290777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:34.290894 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:34.291454 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:34.291471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:34.295116 (+ 3645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":256,"threads_started":1}
W20260811 12:34:34.295534 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:34.286501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:42093 (local address 127.3.204.126:42581)
0811 12:34:34.286718 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:34.286726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:34.287091 (+ 365us) server_negotiation.cc:407] Connection header received
0811 12:34:34.287224 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:34.287228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:34.287294 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:34.287378 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:34.288173 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.288903 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.290155 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.290634 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.290732 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:34.295312 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:42093: BlockingRecv error: recv got EOF from 127.3.204.67:42093 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 11 12:34:34 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:34 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:34.484297 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:34.475005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:34505)
0811 12:34:34.475550 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:34.475568 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:34.475711 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:34.476034 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:34.476039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:34.476507 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:34.476693 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:34.476702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.477764 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.477768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:34.478726 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:34.478739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.479512 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.479519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:34.479695 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:34.480438 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:34.480456 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:34.483937 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
W20260811 12:34:34.484464 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:34.475266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:34505 (local address 127.3.204.126:42581)
0811 12:34:34.475516 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:34.475524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:34.475648 (+ 124us) server_negotiation.cc:407] Connection header received
0811 12:34:34.475836 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:34.475841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:34.475905 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:34.475989 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:34.476840 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.477614 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.478878 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.479327 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.479440 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:34.484210 (+ 4770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:34505: BlockingRecv error: recv got EOF from 127.3.204.65:34505 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 11 12:34:34 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:34 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:34.854599 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:34.844908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:37171)
0811 12:34:34.845498 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:34.845516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:34.845645 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:34.846003 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:34.846007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:34.846460 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:34.846644 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:34.846654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.847934 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.847939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:34.848983 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:34.849000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.849907 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.849915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:34.850066 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:34.850747 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:34.850773 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:34.854338 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":302,"threads_started":1}
W20260811 12:34:34.854947 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:34.845075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:37171 (local address 127.3.204.126:42581)
0811 12:34:34.845242 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:34.845248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:34.845627 (+ 379us) server_negotiation.cc:407] Connection header received
0811 12:34:34.845771 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:34.845775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:34.845849 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:34.845954 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:34.846792 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.847773 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.849140 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:34.849690 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:34.849810 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:34.854628 (+ 4818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:37171: BlockingRecv error: recv got EOF from 127.3.204.66:37171 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 11 12:34:35 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:35 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:35.305742 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:35.296608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:44947)
0811 12:34:35.297207 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:35.297223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:35.297360 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:35.297733 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:35.297737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:35.298237 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:35.298464 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:35.298477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.299527 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.299531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:35.300626 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:35.300640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.301386 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.301393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:35.301554 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:35.302135 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:35.302152 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:35.305496 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":328,"threads_started":1}
W20260811 12:34:35.305959 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:35.296763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44947 (local address 127.3.204.126:42581)
0811 12:34:35.296937 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:35.296942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:35.297353 (+ 411us) server_negotiation.cc:407] Connection header received
0811 12:34:35.297474 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:35.297479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:35.297563 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:35.297669 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:35.298639 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.299392 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.300775 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.301203 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.301351 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:35.305724 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:44947: BlockingRecv error: recv got EOF from 127.3.204.67:44947 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 12:34:35 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:35 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:35.495437 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:35.485496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:44649)
0811 12:34:35.486129 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:35.486147 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:35.486303 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:35.486693 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:35.486698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:35.487168 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:35.487355 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:35.487370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.488778 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.488782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:35.489988 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:35.490003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.490793 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.490797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:35.490921 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:35.491507 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:35.491524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:35.495186 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":337,"threads_started":1}
W20260811 12:34:35.495659 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:35.485594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:44649 (local address 127.3.204.126:42581)
0811 12:34:35.485922 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:35.485930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:35.486253 (+ 323us) server_negotiation.cc:407] Connection header received
0811 12:34:35.486442 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:35.486446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:35.486540 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:35.486636 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:35.487574 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.488638 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.490151 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.490664 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.490764 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:35.495416 (+ 4652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:44649: BlockingRecv error: recv got EOF from 127.3.204.65:44649 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Aug 11 12:34:35 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:35 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:35.864946 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:35.855782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:59637)
0811 12:34:35.856387 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:35.856409 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:35.856562 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:35.856971 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:35.856977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:35.857557 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:35.857747 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:35.857757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.858781 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.858785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:35.859732 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:35.859743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.860477 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.860481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:35.860601 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:35.861179 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:35.861205 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:35.864693 (+ 3488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":339,"threads_started":1}
W20260811 12:34:35.865155 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:35.855886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:59637 (local address 127.3.204.126:42581)
0811 12:34:35.856098 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:35.856105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:35.856504 (+ 399us) server_negotiation.cc:407] Connection header received
0811 12:34:35.856711 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:35.856715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:35.856799 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:35.856907 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:35.857882 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.858646 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.859862 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:35.860358 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:35.860453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:35.864898 (+ 4445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:59637: BlockingRecv error: recv got EOF from 127.3.204.66:59637 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 11 12:34:36 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:36 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:36.319926 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:36.306942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:47331)
0811 12:34:36.307479 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:36.307497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:36.307638 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:36.307996 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:36.308001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:36.308491 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:36.308650 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:36.308660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.309668 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.309673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:36.310629 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:36.310642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.311390 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.311397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:36.311567 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:36.316073 (+ 4506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:36.316127 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:36.319602 (+ 3475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":271,"threads_started":1}
W20260811 12:34:36.320114 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:36.307102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:47331 (local address 127.3.204.126:42581)
0811 12:34:36.307321 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:36.307328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:36.307596 (+ 268us) server_negotiation.cc:407] Connection header received
0811 12:34:36.307766 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:36.307771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:36.307838 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:36.307923 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:36.308802 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.309529 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.310776 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.311236 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.311349 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:36.319828 (+ 8479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:47331: BlockingRecv error: recv got EOF from 127.3.204.67:47331 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 11 12:34:36 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:36 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:36.505843 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:36.496423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:39011)
0811 12:34:36.496976 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:36.496993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:36.497131 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:36.497493 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:36.497498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:36.497910 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:36.498083 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:36.498093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.499127 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.499131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:36.500185 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:36.500199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.500968 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.500974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:36.501148 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:36.501748 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:36.501766 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:36.505456 (+ 3690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":291,"threads_started":1}
W20260811 12:34:36.505942 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:36.496556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:39011 (local address 127.3.204.126:42581)
0811 12:34:36.496803 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:36.496810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:36.497079 (+ 269us) server_negotiation.cc:407] Connection header received
0811 12:34:36.497269 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:36.497274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:36.497338 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:36.497447 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:36.498236 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.498984 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.500332 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.500796 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.500924 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:36.505667 (+ 4743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:39011: BlockingRecv error: recv got EOF from 127.3.204.65:39011 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 11 12:34:36 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:36 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:36.875087 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:36.865873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:52339)
0811 12:34:36.866447 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:36.866463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:36.866607 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:36.866960 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:36.866965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:36.867388 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:36.867555 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:36.867564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.868640 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.868644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:36.869633 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:36.869646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.870345 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.870351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:36.870503 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:36.871116 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:36.871136 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:36.874773 (+ 3637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":295,"threads_started":1}
W20260811 12:34:36.875255 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:36.865958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:52339 (local address 127.3.204.126:42581)
0811 12:34:36.866205 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:36.866211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:36.866553 (+ 342us) server_negotiation.cc:407] Connection header received
0811 12:34:36.866732 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:36.866737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:36.866820 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:36.866906 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:36.867706 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.868493 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.869786 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:36.870186 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:36.870295 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:36.875008 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:52339: BlockingRecv error: recv got EOF from 127.3.204.66:52339 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 11 12:34:37 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:37 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:37.330622 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:37.321087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:44599)
0811 12:34:37.321670 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:37.321687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:37.321808 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:37.322106 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:37.322109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:37.322515 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:37.322690 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:37.322700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.323719 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.323723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:37.324824 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:37.324840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.325795 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.325801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:37.325964 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:37.326711 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:37.326737 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:37.330275 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":306,"threads_started":1}
W20260811 12:34:37.330806 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:37.321237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44599 (local address 127.3.204.126:42581)
0811 12:34:37.321416 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:37.321424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:37.321758 (+ 334us) server_negotiation.cc:407] Connection header received
0811 12:34:37.321903 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:37.321906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:37.321980 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:37.322065 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:37.322829 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.323593 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.324993 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.325601 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.325779 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:37.330537 (+ 4758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:44599: BlockingRecv error: recv got EOF from 127.3.204.67:44599 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 11 12:34:37 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:37 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:37.516109 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:37.507008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:34221)
0811 12:34:37.507544 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:37.507560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:37.507702 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:37.508030 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:37.508035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:37.508535 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:37.508724 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:37.508734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.509731 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.509736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:37.510659 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:37.510671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.511479 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.511485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:37.511663 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:37.512388 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:37.512414 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:37.515840 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
W20260811 12:34:37.516322 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:37.507155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:34221 (local address 127.3.204.126:42581)
0811 12:34:37.507460 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:37.507466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:37.507644 (+ 178us) server_negotiation.cc:407] Connection header received
0811 12:34:37.507830 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:37.507835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:37.507899 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:37.507980 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:37.508879 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.509603 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.510800 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.511324 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.511443 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:37.516046 (+ 4603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:34221: BlockingRecv error: recv got EOF from 127.3.204.65:34221 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 11 12:34:37 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:37 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:37.885442 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:37.876142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:35631)
0811 12:34:37.876634 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:37.876651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:37.876770 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:37.877051 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:37.877054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:37.877500 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:37.877684 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:37.877694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.878707 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.878711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:37.879706 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:37.879720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.880380 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.880385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:37.880554 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:37.881401 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:37.881420 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:37.885135 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
W20260811 12:34:37.885640 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:37.876284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:35631 (local address 127.3.204.126:42581)
0811 12:34:37.876511 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:37.876518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:37.876721 (+ 203us) server_negotiation.cc:407] Connection header received
0811 12:34:37.876869 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:37.876874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:37.876941 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:37.877024 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:37.877825 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.878591 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.879814 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:37.880255 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:37.880343 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:37.885375 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:35631: BlockingRecv error: recv got EOF from 127.3.204.66:35631 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 11 12:34:38 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:38 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:38.368362 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:38.333047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:60587)
0811 12:34:38.334716 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:38.334813 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:38.335282 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:38.336438 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:38.336460 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:38.338346 (+ 1886us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:38.339004 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:38.339088 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.342121 (+ 3033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.342126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:38.345280 (+ 3154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:38.345347 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.348894 (+ 3547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.348972 (+ 78us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:38.349222 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:38.351084 (+ 1862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:38.351245 (+ 161us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:38.367172 (+ 15927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":836,"threads_started":1}
W20260811 12:34:38.369594 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:38.333101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:60587 (local address 127.3.204.126:42581)
0811 12:34:38.333281 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:38.333287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:38.335086 (+ 1799us) server_negotiation.cc:407] Connection header received
0811 12:34:38.335611 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:38.335615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:38.335933 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:38.336256 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:38.339403 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.341699 (+ 2296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.345774 (+ 4075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.348169 (+ 2395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.348760 (+ 591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:38.368476 (+ 19716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:60587: BlockingRecv error: recv got EOF from 127.3.204.67:60587 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 11 12:34:38 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:38 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:38.555480 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:38.518828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:52839)
0811 12:34:38.520842 (+ 2014us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:38.520900 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:38.521414 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:38.522719 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:38.522744 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:38.524488 (+ 1744us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:38.525498 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:38.525561 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.528999 (+ 3438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.529027 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:38.533707 (+ 4680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:38.533818 (+ 111us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.537405 (+ 3587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.537459 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:38.538419 (+ 960us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:38.540864 (+ 2445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:38.540962 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:38.553679 (+ 12717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1586,"thread_start_us":915,"threads_started":1}
W20260811 12:34:38.557536 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:38.519084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:52839 (local address 127.3.204.126:42581)
0811 12:34:38.519676 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:38.519706 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:38.521175 (+ 1469us) server_negotiation.cc:407] Connection header received
0811 12:34:38.521964 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:38.521974 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:38.522282 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:38.522668 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:38.526029 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.528348 (+ 2319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.534388 (+ 6040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.536787 (+ 2399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.537298 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:38.555910 (+ 18612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:52839: BlockingRecv error: recv got EOF from 127.3.204.65:52839 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Aug 11 12:34:38 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:38 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:38.895431 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:38.886392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:55011)
0811 12:34:38.886918 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:38.886935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:38.887050 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:38.887370 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:38.887374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:38.887806 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:38.888006 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:38.888016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.889036 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.889040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:38.890017 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:38.890030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.890786 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.890790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:38.890928 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:38.891489 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:38.891506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:38.895165 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":276,"threads_started":1}
W20260811 12:34:38.895607 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:38.886437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:55011 (local address 127.3.204.126:42581)
0811 12:34:38.886634 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:38.886642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:38.887015 (+ 373us) server_negotiation.cc:407] Connection header received
0811 12:34:38.887147 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:38.887150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:38.887225 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:38.887307 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:38.888167 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.888887 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.890147 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:38.890624 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:38.890722 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:38.895359 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:55011: BlockingRecv error: recv got EOF from 127.3.204.66:55011 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 11 12:34:39 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:39 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:39.379906 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:39.370408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:47145)
0811 12:34:39.370975 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:39.371006 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:39.371126 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:39.371460 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:39.371463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:39.371877 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:39.372110 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:39.372125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.373234 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.373238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:39.374184 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:39.374196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.375067 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.375073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:39.375221 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:39.375853 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:39.375882 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:39.379591 (+ 3709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":324,"threads_started":1}
W20260811 12:34:39.380213 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:39.370512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:47145 (local address 127.3.204.126:42581)
0811 12:34:39.370726 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:39.370735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:39.371089 (+ 354us) server_negotiation.cc:407] Connection header received
0811 12:34:39.371233 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:39.371264 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:39.371338 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:39.371434 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:39.372314 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.373120 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.374359 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.374803 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.374921 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:39.379859 (+ 4938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:47145: BlockingRecv error: recv got EOF from 127.3.204.67:47145 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 11 12:34:39 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:39 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:39.568157 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:39.558168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:60117)
0811 12:34:39.558730 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:39.558747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:39.558885 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:39.559334 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:39.559338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:39.559778 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:39.559948 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:39.559958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.561050 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.561055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:39.562063 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:39.562087 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.562831 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.562836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:39.563051 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:39.563659 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:39.563678 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:39.567870 (+ 4192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":315,"threads_started":1}
W20260811 12:34:39.568296 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:39.558361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:60117 (local address 127.3.204.126:42581)
0811 12:34:39.558568 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:39.558575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:39.558868 (+ 293us) server_negotiation.cc:407] Connection header received
0811 12:34:39.559105 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:39.559110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:39.559189 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:39.559276 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:39.560144 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.560890 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.562237 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.562655 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.562761 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:39.568037 (+ 5276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:60117: BlockingRecv error: recv got EOF from 127.3.204.65:60117 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 11 12:34:39 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:39 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:39.906392 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:39.896852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:55405)
0811 12:34:39.897420 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:39.897440 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:39.897572 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:39.897921 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:39.897925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:39.898380 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:39.898547 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:39.898556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.899637 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.899643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:39.900810 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:39.900826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.901706 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.901713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:39.901896 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:39.902538 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:39.902558 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:39.906027 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":332,"threads_started":1}
W20260811 12:34:39.906508 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:39.897156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:55405 (local address 127.3.204.126:42581)
0811 12:34:39.897385 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:39.897393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:39.897517 (+ 124us) server_negotiation.cc:407] Connection header received
0811 12:34:39.897677 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:39.897681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:39.897747 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:39.897833 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:39.898679 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.899430 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.900972 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:39.901545 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:39.901667 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:39.906273 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:55405: BlockingRecv error: recv got EOF from 127.3.204.66:55405 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 11 12:34:40 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:40 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.67@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:40.390973 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:40.381081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.67:39103)
0811 12:34:40.381632 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:40.381651 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:40.381825 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:40.382189 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:40.382194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:40.382673 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:40.382855 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:40.382867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.384019 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.384024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:40.385015 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:40.385030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.385867 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.385874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:40.386072 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:40.386806 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:40.386826 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:40.390633 (+ 3807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":301,"threads_started":1}
W20260811 12:34:40.391121 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:40.381229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:39103 (local address 127.3.204.126:42581)
0811 12:34:40.381479 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:40.381486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:40.381764 (+ 278us) server_negotiation.cc:407] Connection header received
0811 12:34:40.381959 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:40.381966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:40.382036 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:40.382131 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:40.383029 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.383853 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.385176 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.385706 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.385845 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:40.390899 (+ 5054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.67:39103: BlockingRecv error: recv got EOF from 127.3.204.67:39103 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 11 12:34:40 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:40 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.65@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:40.579082 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:40.569724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.65:34053)
0811 12:34:40.570421 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:40.570437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:40.570571 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:40.571038 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:40.571042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:40.571486 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:40.571679 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:40.571689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.572751 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.572755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:40.574134 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:40.574149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.574905 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.574909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:40.575047 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:40.575605 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:40.575623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:40.578826 (+ 3203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":408,"threads_started":1}
W20260811 12:34:40.579241 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:40.569820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:34053 (local address 127.3.204.126:42581)
0811 12:34:40.570139 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:40.570145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:40.570519 (+ 374us) server_negotiation.cc:407] Connection header received
0811 12:34:40.570736 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:40.570743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:40.570869 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:40.570984 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:40.571830 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.572625 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.574272 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.574774 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.574879 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:40.579019 (+ 4140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.65:34053: BlockingRecv error: recv got EOF from 127.3.204.65:34053 (error 108)
Metrics: {"server-negotiator.queue_time_us":209}
Aug 11 12:34:40 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 12:34:40 dist-test-slave-lgr5 krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.204.66@KRBTEST.COM for oryx/127.3.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 12:34:40.917164 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:40.907755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42581 (local address 127.3.204.66:55935)
0811 12:34:40.908301 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:40.908319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:40.908445 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:40.908827 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:40.908831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:40.909218 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:40.909411 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:40.909420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.910460 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.910464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:40.911448 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:40.911468 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.912378 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.912383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:40.912525 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:40.913217 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:40.913242 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:40.916881 (+ 3639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.204.126:42581: Server oryx/127.3.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":255,"threads_started":1}
W20260811 12:34:40.917280 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:40.907888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:55935 (local address 127.3.204.126:42581)
0811 12:34:40.908069 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:40.908074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:40.908390 (+ 316us) server_negotiation.cc:407] Connection header received
0811 12:34:40.908544 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:40.908548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:40.908632 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:40.908727 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:40.909559 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.910288 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.911586 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:40.912206 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:40.912325 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:40.917084 (+ 4759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.204.66:55935: BlockingRecv error: recv got EOF from 127.3.204.66:55935 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:34:41.173247 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 19208
I20260811 12:34:41.186117 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:41.174467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.3.204.65:35531)
0811 12:34:41.175153 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:41.175161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:41.175188 (+ 27us) server_negotiation.cc:407] Connection header received
0811 12:34:41.175255 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:41.175261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:41.175536 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:41.175709 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:41.177197 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:41.178289 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:41.179530 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:41.180253 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:41.180399 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:41.182972 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:41.182982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:41.182998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:41.183043 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:41.185599 (+ 2556us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.204.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 12:34:41.185900 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58864: Request ticket server kudu/127.3.204.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":373,"threads_started":1}
W20260811 12:34:41.186239 19988 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58864: Request ticket server kudu/127.3.204.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260811 12:34:41.186317 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 19208: Not authorized: Client connection negotiation failed: client connection to 127.3.204.65:35531: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.204.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 12:34:41.186544 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 19208
I20260811 12:34:41.218112 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 19470
I20260811 12:34:41.231389 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:41.219382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.3.204.66:34561)
0811 12:34:41.220064 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:41.220072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:41.220134 (+ 62us) server_negotiation.cc:407] Connection header received
0811 12:34:41.220213 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:41.220220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:41.220472 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:41.220645 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:41.222365 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:41.223506 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:41.224915 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:41.225557 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:41.225670 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:41.228103 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:41.228117 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:41.228137 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:41.228189 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:41.230837 (+ 2648us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.204.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 12:34:41.231130 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50720: Request ticket server kudu/127.3.204.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":365,"threads_started":1}
W20260811 12:34:41.231504 19989 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50720: Request ticket server kudu/127.3.204.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260811 12:34:41.231611 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 19470: Not authorized: Client connection negotiation failed: client connection to 127.3.204.66:34561: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.204.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 12:34:41.231765 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 19470
I20260811 12:34:41.263690 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 19690
I20260811 12:34:41.276762 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:41.264896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.3.204.67:42691)
0811 12:34:41.265592 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:41.265601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:41.265634 (+ 33us) server_negotiation.cc:407] Connection header received
0811 12:34:41.265717 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:41.265727 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:41.266041 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:41.266265 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:41.267586 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:41.269049 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:41.270386 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:41.270973 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:41.271084 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:41.273582 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:41.273591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:41.273607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:41.273648 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:41.276205 (+ 2557us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.204.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 12:34:41.276508 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56376: Request ticket server kudu/127.3.204.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":328,"threads_started":1}
W20260811 12:34:41.276883 19992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56376: Request ticket server kudu/127.3.204.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260811 12:34:41.276950 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 19690: Not authorized: Client connection negotiation failed: client connection to 127.3.204.67:42691: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.204.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 12:34:41.277134 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 19690
I20260811 12:34:41.300556 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 19080
I20260811 12:34:41.302006 19197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:41.421454 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 19080
2026-08-11T12:34:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21724 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-11T12:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:34:41Z Disabled control of system clock
I20260811 12:34:41.513814 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:42109
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:42109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 12:34:41.851233 19998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:41.851548 19998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:41.851655 19998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:41.859053 19998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:41.859176 19998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:41.859222 19998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:41.859304 19998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:41.859370 19998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:41.869243 19998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:42109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:42109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:41.870608 19998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:41.872423 19998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:41.884583 20012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:41.884620 20010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:41.884649 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:41.885087 19998 server_base.cc:1034] running on GCE node
I20260811 12:34:41.885905 19998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:41.887316 19998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:41.888528 19998 hybrid_clock.cc:648] HybridClock initialized: now 1786451681888476 us; error 48 us; skew 500 ppm
I20260811 12:34:41.888883 19998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:34:41.894770 19998 webserver.cc:460] Webserver started at http://127.3.204.126:41211/ using document root <none> and password file <none>
I20260811 12:34:41.895579 19998 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:41.895697 19998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:41.896040 19998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:41.899237 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "0a69d9390f1f4f75bbc6e36656ceba70"
format_stamp: "Formatted at 2026-08-11 12:34:41 on dist-test-slave-lgr5"
server_key: "5f6f7485e33f4435f193f9bdb429c246"
server_key_iv: "8ddbc6e73004decc1b48d5e99a2c07db"
server_key_version: "encryptionkey@0"
I20260811 12:34:41.900038 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "0a69d9390f1f4f75bbc6e36656ceba70"
format_stamp: "Formatted at 2026-08-11 12:34:41 on dist-test-slave-lgr5"
server_key: "5f6f7485e33f4435f193f9bdb429c246"
server_key_iv: "8ddbc6e73004decc1b48d5e99a2c07db"
server_key_version: "encryptionkey@0"
I20260811 12:34:41.905398 19998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 12:34:41.909116 20026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:41.910558 19998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:34:41.910708 19998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "0a69d9390f1f4f75bbc6e36656ceba70"
format_stamp: "Formatted at 2026-08-11 12:34:41 on dist-test-slave-lgr5"
server_key: "5f6f7485e33f4435f193f9bdb429c246"
server_key_iv: "8ddbc6e73004decc1b48d5e99a2c07db"
server_key_version: "encryptionkey@0"
I20260811 12:34:41.910897 19998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:41.922597 19998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:41.923413 19998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:41.923691 19998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:41.940237 19998 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:42109
I20260811 12:34:41.940258 20102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:42109 every 8 connection(s)
I20260811 12:34:41.941592 19998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:34:41.945449 20105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:41.949064 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 19998
I20260811 12:34:41.949237 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:34:41.949865 3889 external_mini_cluster.cc:1428] Setting key 75455eafc9156e1fdbb9d3979e03e86c
I20260811 12:34:41.958027 20105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70: Bootstrap starting.
I20260811 12:34:41.961746 20105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:41.963060 20105 log.cc:824] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:41.966367 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:41.951973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.3.204.126:42109)
0811 12:34:41.953092 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:41.953107 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:41.953163 (+ 56us) server_negotiation.cc:407] Connection header received
0811 12:34:41.956015 (+ 2852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:41.956051 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:41.956705 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:41.957169 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:41.957737 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:41.959727 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:41.961069 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:41.962128 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:41.962334 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:41.963480 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:41.963492 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:41.963511 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:41.964236 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:41.964347 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:41.964369 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:41.964458 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:41.964740 (+ 282us) server_negotiation.cc:299] Negotiation successful
0811 12:34:41.965086 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":319,"threads_started":1}
I20260811 12:34:41.966643 20105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70: No bootstrap required, opened a new log
I20260811 12:34:41.975179 20105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69d9390f1f4f75bbc6e36656ceba70" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42109 } }
I20260811 12:34:41.975683 20105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:41.975791 20105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69d9390f1f4f75bbc6e36656ceba70, State: Initialized, Role: FOLLOWER
I20260811 12:34:41.976457 20105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69d9390f1f4f75bbc6e36656ceba70" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42109 } }
I20260811 12:34:41.976766 20105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:34:41.976910 20105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:34:41.977097 20105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:41.980579 20105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69d9390f1f4f75bbc6e36656ceba70" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42109 } }
I20260811 12:34:41.981199 20105 leader_election.cc:302] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a69d9390f1f4f75bbc6e36656ceba70; no voters:
I20260811 12:34:41.981824 20105 leader_election.cc:288] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:34:41.981954 20114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:41.982661 20114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [term 1 LEADER]: Becoming Leader. State: Replica: 0a69d9390f1f4f75bbc6e36656ceba70, State: Running, Role: LEADER
I20260811 12:34:41.983212 20114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69d9390f1f4f75bbc6e36656ceba70" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42109 } }
I20260811 12:34:41.984041 20105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:41.985936 20119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a69d9390f1f4f75bbc6e36656ceba70. Latest consensus state: current_term: 1 leader_uuid: "0a69d9390f1f4f75bbc6e36656ceba70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69d9390f1f4f75bbc6e36656ceba70" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42109 } } }
I20260811 12:34:41.985936 20115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a69d9390f1f4f75bbc6e36656ceba70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69d9390f1f4f75bbc6e36656ceba70" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42109 } } }
I20260811 12:34:41.986414 20115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:41.991032 20119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70 [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:41.991708 20125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:34:41.996541 20125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:34:42.006353 20125 catalog_manager.cc:1347] Generated new cluster ID: 7e0a8649e377479dbe127ba0b0ed491c
I20260811 12:34:42.006486 20125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:34:42.046396 20125 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:34:42.047858 20125 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:34:42.058804 20125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a69d9390f1f4f75bbc6e36656ceba70: Generated new TSK 0
I20260811 12:34:42.059697 20125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:34:42.073598 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42109
--builtin_ntp_servers=127.3.204.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260811 12:34:42.430543 20158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:42.430842 20158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:42.430963 20158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:42.438524 20158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:42.438639 20158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:42.438776 20158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:34:42.448351 20158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:42.449747 20158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:42.451566 20158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:42.464653 20169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:42.464854 20168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:42.464701 20174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:42.465852 20158 server_base.cc:1034] running on GCE node
I20260811 12:34:42.466550 20158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:42.467566 20158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:42.468767 20158 hybrid_clock.cc:648] HybridClock initialized: now 1786451682468698 us; error 61 us; skew 500 ppm
I20260811 12:34:42.469098 20158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:34:42.471524 20158 webserver.cc:460] Webserver started at http://127.3.204.65:38733/ using document root <none> and password file <none>
I20260811 12:34:42.472322 20158 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:42.472433 20158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:42.472779 20158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:42.475893 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "542da459d370419aaee74016c9667dfa"
format_stamp: "Formatted at 2026-08-11 12:34:42 on dist-test-slave-lgr5"
server_key: "aba2508578f5c466651c93044c7b979c"
server_key_iv: "b89fffff65fdb554ea34b686e6496d42"
server_key_version: "encryptionkey@0"
I20260811 12:34:42.476787 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "542da459d370419aaee74016c9667dfa"
format_stamp: "Formatted at 2026-08-11 12:34:42 on dist-test-slave-lgr5"
server_key: "aba2508578f5c466651c93044c7b979c"
server_key_iv: "b89fffff65fdb554ea34b686e6496d42"
server_key_version: "encryptionkey@0"
I20260811 12:34:42.482511 20158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260811 12:34:42.486438 20183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:42.487897 20158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:34:42.488158 20158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "542da459d370419aaee74016c9667dfa"
format_stamp: "Formatted at 2026-08-11 12:34:42 on dist-test-slave-lgr5"
server_key: "aba2508578f5c466651c93044c7b979c"
server_key_iv: "b89fffff65fdb554ea34b686e6496d42"
server_key_version: "encryptionkey@0"
I20260811 12:34:42.488397 20158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:42.508129 20158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:42.509116 20158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:42.509425 20158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:42.510519 20158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:42.512163 20158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:42.512272 20158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:42.512382 20158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:42.512451 20158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:42.543988 20158 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:42773
I20260811 12:34:42.544034 20361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:42773 every 8 connection(s)
I20260811 12:34:42.545359 20158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:34:42.547513 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 20158
I20260811 12:34:42.547675 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:34:42.548511 3889 external_mini_cluster.cc:1428] Setting key 81887aaf52dfee4c4f36b92e6651bdb6
I20260811 12:34:42.551707 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42109
--builtin_ntp_servers=127.3.204.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 12:34:42.560598 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:42.548058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:46661 (local address 127.3.204.126:42109)
0811 12:34:42.548590 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:42.548596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:42.549950 (+ 1354us) server_negotiation.cc:407] Connection header received
0811 12:34:42.552398 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:42.552403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:42.552500 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:42.552618 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:42.554228 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:42.555019 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:42.557300 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:42.557837 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:42.557952 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:42.559763 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:42.559769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:42.559772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:42.559793 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:42.559836 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:42.559848 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:42.559922 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:42.560201 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 12:34:42.560296 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1}
I20260811 12:34:42.562330 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:42.548743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42109 (local address 127.3.204.65:46661)
0811 12:34:42.549803 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:42.549842 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:42.552007 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:42.552930 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:42.552943 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:42.553080 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:42.553994 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:42.554022 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:42.555190 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:42.555196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:42.557091 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:42.557115 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:42.558089 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:42.558096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:42.558407 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:42.559333 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:34:42.559336 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:42.559396 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:34:42.559522 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:42.559946 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:42.559956 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:42.559964 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 12:34:42.560197 (+ 233us) client_negotiation.cc:240] Negotiation successful
0811 12:34:42.560650 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":319,"threads_started":1}
I20260811 12:34:42.564046 20363 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42109
I20260811 12:34:42.564432 20363 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:42.565441 20363 heartbeater.cc:511] Master 127.3.204.126:42109 requested a full tablet report, sending...
I20260811 12:34:42.567739 20058 ts_manager.cc:194] Registered new tserver with Master: 542da459d370419aaee74016c9667dfa (127.3.204.65:42773)
I20260811 12:34:42.569267 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.65:46661
W20260811 12:34:42.896028 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:42.896395 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:42.896545 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:42.903975 20369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:42.904129 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:42.904282 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:34:42.913642 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:42.915050 20369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:42.917009 20369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:42.930394 20383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:42.931123 20377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:42.931277 20378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:42.932168 20369 server_base.cc:1034] running on GCE node
I20260811 12:34:42.932849 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:42.933818 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:42.935027 20369 hybrid_clock.cc:648] HybridClock initialized: now 1786451682934960 us; error 50 us; skew 500 ppm
I20260811 12:34:42.935359 20369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:34:42.937628 20369 webserver.cc:460] Webserver started at http://127.3.204.66:33083/ using document root <none> and password file <none>
I20260811 12:34:42.938364 20369 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:42.938468 20369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:42.938812 20369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:42.941900 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "4b1b624185a2473eb21e0679986698d1"
format_stamp: "Formatted at 2026-08-11 12:34:42 on dist-test-slave-lgr5"
server_key: "94f88f8232c232f47ac33247881d6909"
server_key_iv: "7f95866543e52cecb44f502e308f95a5"
server_key_version: "encryptionkey@0"
I20260811 12:34:42.942742 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "4b1b624185a2473eb21e0679986698d1"
format_stamp: "Formatted at 2026-08-11 12:34:42 on dist-test-slave-lgr5"
server_key: "94f88f8232c232f47ac33247881d6909"
server_key_iv: "7f95866543e52cecb44f502e308f95a5"
server_key_version: "encryptionkey@0"
I20260811 12:34:42.948405 20369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:34:42.952314 20392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:42.953841 20369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:34:42.954008 20369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "4b1b624185a2473eb21e0679986698d1"
format_stamp: "Formatted at 2026-08-11 12:34:42 on dist-test-slave-lgr5"
server_key: "94f88f8232c232f47ac33247881d6909"
server_key_iv: "7f95866543e52cecb44f502e308f95a5"
server_key_version: "encryptionkey@0"
I20260811 12:34:42.954224 20369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:42.976274 20369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:42.977324 20369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:42.977648 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:42.978698 20369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:42.980347 20369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:42.980463 20369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:42.980578 20369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:42.980649 20369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:43.011417 20369 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:38699
I20260811 12:34:43.011498 20568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:38699 every 8 connection(s)
I20260811 12:34:43.012773 20369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:34:43.013425 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 20369
I20260811 12:34:43.013633 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:34:43.014242 3889 external_mini_cluster.cc:1428] Setting key bed2a5a818e818de50e9186da2374323
I20260811 12:34:43.017102 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42109
--builtin_ntp_servers=127.3.204.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 12:34:43.025817 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:43.015498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:53065 (local address 127.3.204.126:42109)
0811 12:34:43.015711 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:43.015718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:43.017499 (+ 1781us) server_negotiation.cc:407] Connection header received
0811 12:34:43.019332 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:43.019338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:43.019429 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:43.019518 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:43.020716 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.021441 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.023031 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.023428 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.023544 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:43.025005 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:43.025011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:43.025014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:43.025034 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:43.025066 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:43.025078 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:43.025154 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:43.025395 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 12:34:43.025506 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 12:34:43.027137 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:43.016208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42109 (local address 127.3.204.66:53065)
0811 12:34:43.017311 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:43.017381 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:43.019049 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:43.019770 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:43.019786 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:43.019884 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:43.020541 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:43.020561 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.021611 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.021615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:43.022880 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:43.022894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.023665 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.023674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:43.023911 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:43.024660 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:34:43.024663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:43.024708 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:34:43.024800 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:43.025170 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:43.025182 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:43.025190 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 12:34:43.025345 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 12:34:43.025805 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":341,"threads_started":1}
I20260811 12:34:43.028689 20569 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42109
I20260811 12:34:43.029057 20569 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:43.030043 20569 heartbeater.cc:511] Master 127.3.204.126:42109 requested a full tablet report, sending...
I20260811 12:34:43.031749 20058 ts_manager.cc:194] Registered new tserver with Master: 4b1b624185a2473eb21e0679986698d1 (127.3.204.66:38699)
I20260811 12:34:43.032601 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.66:53065
W20260811 12:34:43.368722 20576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:43.369021 20576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:43.369131 20576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:43.376662 20576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:43.376773 20576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:43.376914 20576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:34:43.386300 20576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:43.387687 20576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:43.390023 20576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:43.403735 20584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:43.403818 20586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:43.403789 20581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:43.405118 20576 server_base.cc:1034] running on GCE node
I20260811 12:34:43.405829 20576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:43.406906 20576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:43.408175 20576 hybrid_clock.cc:648] HybridClock initialized: now 1786451683408114 us; error 45 us; skew 500 ppm
I20260811 12:34:43.408557 20576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:34:43.411074 20576 webserver.cc:460] Webserver started at http://127.3.204.67:41561/ using document root <none> and password file <none>
I20260811 12:34:43.411849 20576 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:43.411955 20576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:43.412346 20576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:43.415429 20576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "280c3bbbf2cb4fcebad97b8a936ebb33"
format_stamp: "Formatted at 2026-08-11 12:34:43 on dist-test-slave-lgr5"
server_key: "a6f8f5740ddf018134f3240f9bf23916"
server_key_iv: "2255065295f1bb06618a940bf99ec43e"
server_key_version: "encryptionkey@0"
I20260811 12:34:43.416298 20576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "280c3bbbf2cb4fcebad97b8a936ebb33"
format_stamp: "Formatted at 2026-08-11 12:34:43 on dist-test-slave-lgr5"
server_key: "a6f8f5740ddf018134f3240f9bf23916"
server_key_iv: "2255065295f1bb06618a940bf99ec43e"
server_key_version: "encryptionkey@0"
I20260811 12:34:43.421550 20576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 12:34:43.425097 20598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:43.426676 20576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:34:43.426860 20576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "280c3bbbf2cb4fcebad97b8a936ebb33"
format_stamp: "Formatted at 2026-08-11 12:34:43 on dist-test-slave-lgr5"
server_key: "a6f8f5740ddf018134f3240f9bf23916"
server_key_iv: "2255065295f1bb06618a940bf99ec43e"
server_key_version: "encryptionkey@0"
I20260811 12:34:43.427079 20576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:43.450338 20576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:43.451351 20576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:43.451767 20576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:43.452884 20576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:43.454448 20576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:43.454536 20576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:43.454623 20576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:43.454676 20576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:43.487602 20576 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:40933
I20260811 12:34:43.487655 20779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:40933 every 8 connection(s)
I20260811 12:34:43.489027 20576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:34:43.498005 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 20576
I20260811 12:34:43.498167 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:34:43.498627 3889 external_mini_cluster.cc:1428] Setting key 8cd2df5e27f52bab1ed90e25b1d8133c
I20260811 12:34:43.502694 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:43.491911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:57429 (local address 127.3.204.126:42109)
0811 12:34:43.492138 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:43.492143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:43.493806 (+ 1663us) server_negotiation.cc:407] Connection header received
0811 12:34:43.495862 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:43.495867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:43.495944 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:43.496037 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:43.497238 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.498021 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.499792 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.500312 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.500462 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:43.501983 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:43.501988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:43.501991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:43.502009 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:43.502042 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:43.502054 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:43.502125 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:43.502325 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 12:34:43.502447 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 12:34:43.503751 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:43.492583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42109 (local address 127.3.204.67:57429)
0811 12:34:43.493639 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:43.493687 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:43.495554 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:43.496330 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:43.496338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:43.496438 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:43.497049 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:43.497068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.498204 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.498208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:43.499647 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:43.499661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.500582 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.500598 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:43.500838 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:43.501602 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:34:43.501606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:43.501654 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:34:43.501771 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:43.502133 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:43.502141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:43.502146 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 12:34:43.502293 (+ 147us) client_negotiation.cc:240] Negotiation successful
0811 12:34:43.502674 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":334,"threads_started":1}
I20260811 12:34:43.505232 20780 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42109
I20260811 12:34:43.505556 20780 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:43.506364 20780 heartbeater.cc:511] Master 127.3.204.126:42109 requested a full tablet report, sending...
I20260811 12:34:43.507979 20058 ts_manager.cc:194] Registered new tserver with Master: 280c3bbbf2cb4fcebad97b8a936ebb33 (127.3.204.67:40933)
I20260811 12:34:43.508872 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.67:57429
I20260811 12:34:43.514369 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 12:34:43.520727 20366 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:43.518963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.3.204.126:42109)
0811 12:34:43.519133 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:43.519140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:43.519637 (+ 497us) server_negotiation.cc:407] Connection header received
0811 12:34:43.519809 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:43.519815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:43.519902 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:43.519998 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:43.520460 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54362: BlockingRecv error: recv got EOF from 127.0.0.1:54362 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 12:34:43.521755 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 20158
I20260811 12:34:43.529553 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:43.522594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.3.204.65:42773)
0811 12:34:43.523079 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:43.523087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:43.523113 (+ 26us) server_negotiation.cc:407] Connection header received
0811 12:34:43.523176 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:43.523182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:43.523386 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:43.523525 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:43.523809 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.524852 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.526262 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.526845 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.526946 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:43.528111 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:43.528125 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:43.528144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:43.528820 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:43.528869 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:43.528896 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:43.528995 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:43.529108 (+ 113us) server_negotiation.cc:299] Negotiation successful
0811 12:34:43.529243 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260811 12:34:43.530874 20353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:43.611737 20363 heartbeater.cc:503] Master 127.3.204.126:42109 was elected leader, sending a full tablet report...
I20260811 12:34:43.613832 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 20158
I20260811 12:34:43.636420 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 20369
I20260811 12:34:43.644546 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:43.637353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.3.204.66:38699)
0811 12:34:43.638040 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:43.638047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:43.638079 (+ 32us) server_negotiation.cc:407] Connection header received
0811 12:34:43.638147 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:43.638154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:43.638428 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:43.638576 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:43.639016 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.640043 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.641311 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.641885 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.641987 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:43.643035 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:43.643043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:43.643058 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:43.643692 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:43.643743 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:43.643765 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:43.643857 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:43.643990 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 12:34:43.644155 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":346,"threads_started":1}
I20260811 12:34:43.645840 20558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:43.727826 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 20369
I20260811 12:34:43.753695 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 20576
I20260811 12:34:43.762270 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:43.754720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.3.204.67:40933)
0811 12:34:43.755394 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:43.755403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:43.755435 (+ 32us) server_negotiation.cc:407] Connection header received
0811 12:34:43.755510 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:43.755517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:43.755740 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:43.755897 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:43.756422 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.757495 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.758765 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:43.759386 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:43.759522 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:43.760725 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:43.760734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:43.760750 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:43.761381 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:43.761435 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:43.761456 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:43.761574 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:43.761757 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 12:34:43.761906 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":360,"threads_started":1}
I20260811 12:34:43.763628 20775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:43.845496 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 20576
I20260811 12:34:43.871070 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 19998
I20260811 12:34:43.872396 20098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:43.983695 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 19998
2026-08-11T12:34:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2558 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-11T12:34:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:34:44Z Disabled control of system clock
I20260811 12:34:44.072935 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:40989
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 12:34:44.415459 20815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:44.415789 20815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:44.415903 20815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:44.423312 20815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:44.423445 20815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:44.423491 20815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:44.423573 20815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:44.423629 20815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:44.433192 20815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:40989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:44.434554 20815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:44.436441 20815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:44.448243 20822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:44.448184 20826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:44.448215 20829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:44.448927 20815 server_base.cc:1034] running on GCE node
I20260811 12:34:44.449785 20815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:44.451306 20815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:44.452517 20815 hybrid_clock.cc:648] HybridClock initialized: now 1786451684452471 us; error 34 us; skew 500 ppm
I20260811 12:34:44.452874 20815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:34:44.455307 20815 webserver.cc:460] Webserver started at http://127.3.204.126:46351/ using document root <none> and password file <none>
I20260811 12:34:44.456063 20815 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:44.456204 20815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:44.456554 20815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:44.459627 20815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "79df656ca0cc43b895e2544358866146"
format_stamp: "Formatted at 2026-08-11 12:34:44 on dist-test-slave-lgr5"
server_key: "9afcf8334f6d866ecf82ca7b4b5de2e0"
server_key_iv: "81f9ed247839d156254d864a444cd851"
server_key_version: "encryptionkey@0"
I20260811 12:34:44.460490 20815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "79df656ca0cc43b895e2544358866146"
format_stamp: "Formatted at 2026-08-11 12:34:44 on dist-test-slave-lgr5"
server_key: "9afcf8334f6d866ecf82ca7b4b5de2e0"
server_key_iv: "81f9ed247839d156254d864a444cd851"
server_key_version: "encryptionkey@0"
I20260811 12:34:44.465950 20815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260811 12:34:44.469754 20838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:44.471196 20815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:34:44.471377 20815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "79df656ca0cc43b895e2544358866146"
format_stamp: "Formatted at 2026-08-11 12:34:44 on dist-test-slave-lgr5"
server_key: "9afcf8334f6d866ecf82ca7b4b5de2e0"
server_key_iv: "81f9ed247839d156254d864a444cd851"
server_key_version: "encryptionkey@0"
I20260811 12:34:44.471601 20815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:44.477843 20815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:44.478875 20815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:44.479182 20815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:44.498594 20815 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:40989
I20260811 12:34:44.498613 20924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:40989 every 8 connection(s)
I20260811 12:34:44.500125 20815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:34:44.504505 20925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:44.509690 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 20815
I20260811 12:34:44.509922 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:34:44.510564 3889 external_mini_cluster.cc:1428] Setting key b0d6d2196547ac44e5a8e0516177c8ca
I20260811 12:34:44.516858 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146: Bootstrap starting.
I20260811 12:34:44.520988 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:44.512858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.3.204.126:40989)
0811 12:34:44.514337 (+ 1479us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:44.514353 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:44.514407 (+ 54us) server_negotiation.cc:407] Connection header received
0811 12:34:44.516762 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:44.516795 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:44.517385 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:44.517849 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:44.518919 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:44.518928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:44.518945 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:44.518968 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:44.519056 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:44.519081 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:44.519320 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:44.519447 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 12:34:44.519716 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":458,"threads_started":1}
I20260811 12:34:44.521088 20925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:44.522224 20925 log.cc:824] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:44.525727 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146: No bootstrap required, opened a new log
I20260811 12:34:44.533181 20925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df656ca0cc43b895e2544358866146" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40989 } }
I20260811 12:34:44.533615 20925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:44.533723 20925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79df656ca0cc43b895e2544358866146, State: Initialized, Role: FOLLOWER
I20260811 12:34:44.534397 20925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df656ca0cc43b895e2544358866146" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40989 } }
I20260811 12:34:44.534617 20925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:34:44.534718 20925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:34:44.534849 20925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:44.537117 20925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df656ca0cc43b895e2544358866146" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40989 } }
I20260811 12:34:44.537598 20925 leader_election.cc:302] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79df656ca0cc43b895e2544358866146; no voters:
I20260811 12:34:44.538195 20925 leader_election.cc:288] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:34:44.538371 20938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:44.539168 20938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [term 1 LEADER]: Becoming Leader. State: Replica: 79df656ca0cc43b895e2544358866146, State: Running, Role: LEADER
I20260811 12:34:44.539734 20938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df656ca0cc43b895e2544358866146" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40989 } }
I20260811 12:34:44.540396 20925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:44.542526 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79df656ca0cc43b895e2544358866146. Latest consensus state: current_term: 1 leader_uuid: "79df656ca0cc43b895e2544358866146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df656ca0cc43b895e2544358866146" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40989 } } }
I20260811 12:34:44.542860 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:44.543716 20947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:34:44.543577 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79df656ca0cc43b895e2544358866146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df656ca0cc43b895e2544358866146" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40989 } } }
I20260811 12:34:44.544503 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146 [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:44.551937 20947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:34:44.565400 20947 catalog_manager.cc:1347] Generated new cluster ID: ee56d47a4214445a90f652c1df21514a
I20260811 12:34:44.565557 20947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:34:44.587805 20947 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:34:44.588428 20947 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:34:44.610806 20947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79df656ca0cc43b895e2544358866146: Generated new TSK 0
I20260811 12:34:44.611784 20947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:34:44.624567 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40989
--builtin_ntp_servers=127.3.204.84:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 12:34:44.971451 20972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:44.971733 20972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:44.971835 20972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:44.979292 20972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:44.979403 20972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:44.979526 20972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:34:44.988982 20972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:44.990321 20972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:44.992218 20972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:45.006122 20989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:45.006122 20987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:45.006371 20991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:45.007030 20972 server_base.cc:1034] running on GCE node
I20260811 12:34:45.007697 20972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:45.008842 20972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:45.010066 20972 hybrid_clock.cc:648] HybridClock initialized: now 1786451685009951 us; error 101 us; skew 500 ppm
I20260811 12:34:45.010505 20972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:34:45.013399 20972 webserver.cc:460] Webserver started at http://127.3.204.65:44285/ using document root <none> and password file <none>
I20260811 12:34:45.014153 20972 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:45.014259 20972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:45.014580 20972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:45.017764 20972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "377bc8407f12477b9617c1cade175e58"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "e24149bb5f9ee1250ca3dc646aece92e"
server_key_iv: "a2e548717d9d86360b00fcb27231ab94"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.018589 20972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "377bc8407f12477b9617c1cade175e58"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "e24149bb5f9ee1250ca3dc646aece92e"
server_key_iv: "a2e548717d9d86360b00fcb27231ab94"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.024151 20972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:34:45.027738 20998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.029187 20972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:34:45.029332 20972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "377bc8407f12477b9617c1cade175e58"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "e24149bb5f9ee1250ca3dc646aece92e"
server_key_iv: "a2e548717d9d86360b00fcb27231ab94"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.029536 20972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:45.035285 20972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:45.036165 20972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:45.036406 20972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:45.037355 20972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:45.038841 20972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:45.038928 20972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.039012 20972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:45.039065 20972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.071557 20972 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:34719
I20260811 12:34:45.071612 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:34719 every 8 connection(s)
I20260811 12:34:45.072994 20972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:34:45.083210 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 20972
I20260811 12:34:45.083463 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:34:45.083772 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:45.075753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:37763 (local address 127.3.204.126:40989)
0811 12:34:45.076273 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:45.076278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:45.079540 (+ 3262us) server_negotiation.cc:407] Connection header received
0811 12:34:45.081330 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:45.081336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:45.081405 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:45.081515 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:45.082976 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:45.082981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:45.082984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:45.083002 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:45.083063 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:45.083076 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:45.083187 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:45.083436 (+ 249us) server_negotiation.cc:299] Negotiation successful
0811 12:34:45.083497 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":299,"threads_started":1}
I20260811 12:34:45.084285 3889 external_mini_cluster.cc:1428] Setting key c86b639175b4cb0f2689f64e40c6c304
I20260811 12:34:45.085100 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:45.076438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40989 (local address 127.3.204.65:37763)
0811 12:34:45.079345 (+ 2907us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:45.079394 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:45.081039 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:45.081783 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:45.081792 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:45.081891 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:45.082588 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:34:45.082590 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:45.082630 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:34:45.082742 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:45.083187 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:45.083199 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:45.083207 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 12:34:45.083503 (+ 296us) client_negotiation.cc:240] Negotiation successful
0811 12:34:45.083798 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2289,"thread_start_us":2091,"threads_started":1}
I20260811 12:34:45.086663 21199 heartbeater.cc:347] Connected to a master server at 127.3.204.126:40989
I20260811 12:34:45.086963 21199 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:45.087350 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40989
--builtin_ntp_servers=127.3.204.84:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 12:34:45.087604 21199 heartbeater.cc:511] Master 127.3.204.126:40989 requested a full tablet report, sending...
I20260811 12:34:45.089828 20869 ts_manager.cc:194] Registered new tserver with Master: 377bc8407f12477b9617c1cade175e58 (127.3.204.65:34719)
W20260811 12:34:45.430651 21204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:45.430962 21204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:45.431066 21204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:45.438433 21204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:45.438545 21204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:45.438654 21204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:34:45.448215 21204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:45.449540 21204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:45.451346 21204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:45.465610 21212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:45.465549 21219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:45.465636 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:45.466327 21204 server_base.cc:1034] running on GCE node
I20260811 12:34:45.467005 21204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:45.468057 21204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:45.469296 21204 hybrid_clock.cc:648] HybridClock initialized: now 1786451685469241 us; error 49 us; skew 500 ppm
I20260811 12:34:45.469630 21204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:34:45.472004 21204 webserver.cc:460] Webserver started at http://127.3.204.66:37257/ using document root <none> and password file <none>
I20260811 12:34:45.472743 21204 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:45.472853 21204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:45.473217 21204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:45.476590 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "e15f8ec3d2d0448ea2b513999c34fbd9"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "809657f054a18df332fd4371d5d9fc83"
server_key_iv: "43da9e9b13d34f4df9a7d6bb353b637a"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.477432 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "e15f8ec3d2d0448ea2b513999c34fbd9"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "809657f054a18df332fd4371d5d9fc83"
server_key_iv: "43da9e9b13d34f4df9a7d6bb353b637a"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.482919 21204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 12:34:45.486958 21229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.488520 21204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:34:45.488693 21204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "e15f8ec3d2d0448ea2b513999c34fbd9"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "809657f054a18df332fd4371d5d9fc83"
server_key_iv: "43da9e9b13d34f4df9a7d6bb353b637a"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.488917 21204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:45.495048 21204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:45.496023 21204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:45.496330 21204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:45.497349 21204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:45.499302 21204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:45.499418 21204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.499538 21204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:45.499611 21204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.533744 21204 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:42857
I20260811 12:34:45.533798 21433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:42857 every 8 connection(s)
I20260811 12:34:45.535274 21204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:34:45.535729 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 21204
I20260811 12:34:45.535943 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:34:45.536540 3889 external_mini_cluster.cc:1428] Setting key aabc7dda7e8ba7d918d7695bfff3d6a9
I20260811 12:34:45.539944 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40989
--builtin_ntp_servers=127.3.204.84:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 12:34:45.545758 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:45.538595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:34997 (local address 127.3.204.126:40989)
0811 12:34:45.538842 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:45.538849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:45.540753 (+ 1904us) server_negotiation.cc:407] Connection header received
0811 12:34:45.543152 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:45.543158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:45.543222 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:45.543318 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:45.545020 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:45.545025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:45.545029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:45.545047 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:45.545087 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:45.545100 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:45.545209 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:45.545460 (+ 251us) server_negotiation.cc:299] Negotiation successful
0811 12:34:45.545520 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260811 12:34:45.547145 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:45.539270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40989 (local address 127.3.204.66:34997)
0811 12:34:45.540552 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:45.540602 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:45.542806 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:45.543620 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:45.543633 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:45.543769 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:45.544573 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:34:45.544576 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:45.544632 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:34:45.544753 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:45.545197 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:45.545207 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:45.545214 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 12:34:45.545556 (+ 342us) client_negotiation.cc:240] Negotiation successful
0811 12:34:45.545863 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":339,"threads_started":1}
I20260811 12:34:45.548657 21435 heartbeater.cc:347] Connected to a master server at 127.3.204.126:40989
I20260811 12:34:45.548981 21435 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:45.549619 21435 heartbeater.cc:511] Master 127.3.204.126:40989 requested a full tablet report, sending...
I20260811 12:34:45.551165 20869 ts_manager.cc:194] Registered new tserver with Master: e15f8ec3d2d0448ea2b513999c34fbd9 (127.3.204.66:42857)
W20260811 12:34:45.878159 21438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:45.878484 21438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:45.878625 21438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:45.885928 21438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:45.886050 21438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:45.886198 21438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:34:45.895524 21438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:45.896901 21438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:45.898782 21438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:45.912418 21446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:45.912444 21448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:45.912562 21450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:45.913142 21438 server_base.cc:1034] running on GCE node
I20260811 12:34:45.913796 21438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:45.914765 21438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:45.915982 21438 hybrid_clock.cc:648] HybridClock initialized: now 1786451685915916 us; error 51 us; skew 500 ppm
I20260811 12:34:45.916329 21438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:34:45.918661 21438 webserver.cc:460] Webserver started at http://127.3.204.67:33403/ using document root <none> and password file <none>
I20260811 12:34:45.919371 21438 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:45.919480 21438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:45.919811 21438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:45.922787 21438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "a962157f23d54fafa61110ea355454e6"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "ee4409647a79eee56082ebd80a4957c8"
server_key_iv: "3cea0713df8ee37f8bd9e0fa1c844177"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.923661 21438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "a962157f23d54fafa61110ea355454e6"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "ee4409647a79eee56082ebd80a4957c8"
server_key_iv: "3cea0713df8ee37f8bd9e0fa1c844177"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.929185 21438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260811 12:34:45.932853 21460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.934428 21438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:34:45.934587 21438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "a962157f23d54fafa61110ea355454e6"
format_stamp: "Formatted at 2026-08-11 12:34:45 on dist-test-slave-lgr5"
server_key: "ee4409647a79eee56082ebd80a4957c8"
server_key_iv: "3cea0713df8ee37f8bd9e0fa1c844177"
server_key_version: "encryptionkey@0"
I20260811 12:34:45.934805 21438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:45.940763 21438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:45.941623 21438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:45.941869 21438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:45.942770 21438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:45.944192 21438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:45.944288 21438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.944401 21438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:45.944473 21438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:45.980748 21438 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:36743
I20260811 12:34:45.980803 21646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:36743 every 8 connection(s)
I20260811 12:34:45.982293 21438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:34:45.988978 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 21438
I20260811 12:34:45.989181 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:34:45.989734 3889 external_mini_cluster.cc:1428] Setting key c46e234e5053c4cf4aa8c1f220637de2
I20260811 12:34:45.990677 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:45.984906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:52321 (local address 127.3.204.126:40989)
0811 12:34:45.985143 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:45.985149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:45.986581 (+ 1432us) server_negotiation.cc:407] Connection header received
0811 12:34:45.988240 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:45.988245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:45.988318 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:45.988396 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:45.989943 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:45.989949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:45.989952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:45.989972 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:45.990005 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:45.990017 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:45.990130 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:45.990369 (+ 239us) server_negotiation.cc:299] Negotiation successful
0811 12:34:45.990437 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:34:45.991734 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:45.985513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40989 (local address 127.3.204.67:52321)
0811 12:34:45.986413 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:45.986459 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:45.987948 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:45.988625 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:45.988637 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:45.988760 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:45.989534 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:34:45.989537 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:45.989590 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:34:45.989704 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:45.990120 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:45.990130 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:45.990139 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 12:34:45.990395 (+ 256us) client_negotiation.cc:240] Negotiation successful
0811 12:34:45.990638 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":258,"threads_started":1}
I20260811 12:34:45.993125 21649 heartbeater.cc:347] Connected to a master server at 127.3.204.126:40989
I20260811 12:34:45.993432 21649 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:45.994033 21649 heartbeater.cc:511] Master 127.3.204.126:40989 requested a full tablet report, sending...
I20260811 12:34:45.995445 20871 ts_manager.cc:194] Registered new tserver with Master: a962157f23d54fafa61110ea355454e6 (127.3.204.67:36743)
I20260811 12:34:46.005527 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 12:34:46.011754 21201 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:34:46.010252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.3.204.126:40989)
0811 12:34:46.010463 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:46.010470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:46.010887 (+ 417us) server_negotiation.cc:407] Connection header received
0811 12:34:46.011034 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:46.011038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:46.011093 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:46.011174 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:46.011521 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55182: BlockingRecv error: recv got EOF from 127.0.0.1:55182 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 12:34:46.014375 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 20972
I20260811 12:34:46.018412 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:46.015649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.3.204.65:34719)
0811 12:34:46.016251 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:46.016258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:46.016285 (+ 27us) server_negotiation.cc:407] Connection header received
0811 12:34:46.016349 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:46.016355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:46.016571 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:46.016716 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:46.017693 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:46.017700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:46.017712 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:46.017730 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:46.017770 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:46.017783 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:46.017907 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:46.018030 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 12:34:46.018135 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":261,"threads_started":1}
I20260811 12:34:46.019639 21190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:46.099602 21199 heartbeater.cc:503] Master 127.3.204.126:40989 was elected leader, sending a full tablet report...
I20260811 12:34:46.106305 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 20972
I20260811 12:34:46.132002 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 21204
I20260811 12:34:46.135869 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:46.133173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.3.204.66:42857)
0811 12:34:46.133753 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:46.133760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:46.133789 (+ 29us) server_negotiation.cc:407] Connection header received
0811 12:34:46.133871 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:46.133877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:46.134104 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:46.134235 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:46.135173 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:46.135180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:46.135193 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:46.135211 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:46.135253 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:46.135267 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:46.135386 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:46.135442 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 12:34:46.135531 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":295,"threads_started":1}
I20260811 12:34:46.137430 21425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:46.222261 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 21204
I20260811 12:34:46.242571 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 21438
I20260811 12:34:46.246848 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:46.243575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.3.204.67:36743)
0811 12:34:46.244359 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:46.244366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:46.244400 (+ 34us) server_negotiation.cc:407] Connection header received
0811 12:34:46.244474 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:46.244480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:46.244704 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:46.244855 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:46.246060 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:46.246066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:46.246078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:34:46.246096 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:46.246138 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:34:46.246153 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:46.246315 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:46.246425 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 12:34:46.246539 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":315,"threads_started":1}
I20260811 12:34:46.248200 21640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:46.329468 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 21438
I20260811 12:34:46.351982 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 20815
I20260811 12:34:46.353461 20919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:46.454955 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 20815
2026-08-11T12:34:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2467 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:34:46 dist-test-slave-lgr5 krb5kdc[21681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:34:46 dist-test-slave-lgr5 krb5kdc[21681](info): set up 2 sockets
Aug 11 12:34:46 dist-test-slave-lgr5 krb5kdc[21681](info): commencing operation
krb5kdc: starting...
W20260811 12:34:47.712502 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:34:47 dist-test-slave-lgr5 krb5kdc[21681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:34:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:34:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:34:47.994946 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:42729
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:48.342384 21708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:48.342720 21708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:48.342830 21708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:48.350533 21708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:34:48.350666 21708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:48.350731 21708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:48.350796 21708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:34:48.350847 21708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:34:48.361099 21708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:48.362571 21708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:48.363920 21708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:48.375854 21722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:48.375919 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:48.376410 21724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:48.377079 21708 server_base.cc:1034] running on GCE node
I20260811 12:34:48.377930 21708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:48.379390 21708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:48.380594 21708 hybrid_clock.cc:648] HybridClock initialized: now 1786451688380529 us; error 49 us; skew 500 ppm
I20260811 12:34:48.381033 21708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:48 dist-test-slave-lgr5 krb5kdc[21681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451688, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:48.389226 21708 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:34:48.390882 21708 webserver.cc:460] Webserver started at http://127.3.204.126:41407/ using document root <none> and password file <none>
I20260811 12:34:48.391629 21708 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:48.391757 21708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:48.392110 21708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:48.395390 21708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "1c14535dd6ff4e289afde6e4972da168"
format_stamp: "Formatted at 2026-08-11 12:34:48 on dist-test-slave-lgr5"
server_key: "0a5a6e3d0a184ea9ee00f1b682f972ab"
server_key_iv: "5461f28d3ef63ecf51cf05f9487eae0c"
server_key_version: "encryptionkey@0"
I20260811 12:34:48.396224 21708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "1c14535dd6ff4e289afde6e4972da168"
format_stamp: "Formatted at 2026-08-11 12:34:48 on dist-test-slave-lgr5"
server_key: "0a5a6e3d0a184ea9ee00f1b682f972ab"
server_key_iv: "5461f28d3ef63ecf51cf05f9487eae0c"
server_key_version: "encryptionkey@0"
I20260811 12:34:48.401554 21708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 12:34:48.405083 21736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:48.406476 21708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:34:48.406664 21708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "1c14535dd6ff4e289afde6e4972da168"
format_stamp: "Formatted at 2026-08-11 12:34:48 on dist-test-slave-lgr5"
server_key: "0a5a6e3d0a184ea9ee00f1b682f972ab"
server_key_iv: "5461f28d3ef63ecf51cf05f9487eae0c"
server_key_version: "encryptionkey@0"
I20260811 12:34:48.406862 21708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:48.419971 21708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:48.424274 21708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:48.424559 21708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:48.442687 21708 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:42729
I20260811 12:34:48.442708 21823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:42729 every 8 connection(s)
I20260811 12:34:48.444330 21708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:34:48.448421 21825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:48.454013 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 21708
I20260811 12:34:48.454264 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:34:48.455044 3889 external_mini_cluster.cc:1428] Setting key 2070441720326483c42adb9ca8d35881
I20260811 12:34:48.460632 21825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168: Bootstrap starting.
I20260811 12:34:48.463814 21825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:48.464946 21825 log.cc:824] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:48.468506 21825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168: No bootstrap required, opened a new log
Aug 11 12:34:48 dist-test-slave-lgr5 krb5kdc[21681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:34:48.478663 21825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c14535dd6ff4e289afde6e4972da168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42729 } }
I20260811 12:34:48.479177 21825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:48.479293 21825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c14535dd6ff4e289afde6e4972da168, State: Initialized, Role: FOLLOWER
I20260811 12:34:48.480037 21825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c14535dd6ff4e289afde6e4972da168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42729 } }
I20260811 12:34:48.480365 21825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:34:48.480521 21825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:34:48.480720 21825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:48.481171 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:48.457421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.3.204.126:42729)
0811 12:34:48.458387 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:48.458404 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:48.458452 (+ 48us) server_negotiation.cc:407] Connection header received
0811 12:34:48.460214 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:48.460244 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:48.460669 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:48.461066 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:48.462771 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:48.464176 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:48.465983 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:48.466948 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:48.467181 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:48.471719 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:48.471731 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:48.471752 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:48.471803 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:48.476059 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:48.476922 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:48.476935 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:48.476946 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:48.477200 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:48.477624 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:48.477630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:48.477633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:48.478425 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:48.478759 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:48.479189 (+ 430us) server_negotiation.cc:299] Negotiation successful
0811 12:34:48.479683 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":423,"threads_started":1}
I20260811 12:34:48.483839 21825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c14535dd6ff4e289afde6e4972da168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42729 } }
I20260811 12:34:48.484462 21825 leader_election.cc:302] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c14535dd6ff4e289afde6e4972da168; no voters:
I20260811 12:34:48.485123 21825 leader_election.cc:288] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:34:48.485253 21832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:48.485911 21832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [term 1 LEADER]: Becoming Leader. State: Replica: 1c14535dd6ff4e289afde6e4972da168, State: Running, Role: LEADER
I20260811 12:34:48.486455 21832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c14535dd6ff4e289afde6e4972da168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42729 } }
I20260811 12:34:48.487175 21825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:34:48.489396 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c14535dd6ff4e289afde6e4972da168. Latest consensus state: current_term: 1 leader_uuid: "1c14535dd6ff4e289afde6e4972da168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c14535dd6ff4e289afde6e4972da168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42729 } } }
I20260811 12:34:48.489790 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:48.490515 21834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c14535dd6ff4e289afde6e4972da168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c14535dd6ff4e289afde6e4972da168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 42729 } } }
I20260811 12:34:48.490942 21834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168 [sys.catalog]: This master's current role is: LEADER
I20260811 12:34:48.493247 21844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:34:48.498824 21844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:34:48.508152 21844 catalog_manager.cc:1347] Generated new cluster ID: 39a83808fffc4a9ead0466063d9e4d33
I20260811 12:34:48.508283 21844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:34:48.523712 21844 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:34:48.525784 21844 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:34:48.537824 21844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168: Generated new TSK 0
I20260811 12:34:48.539034 21844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:34:48.612689 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42729
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:48.955029 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:48.955330 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:48.955437 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:48.962700 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:48.962821 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:48.962940 21877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:34:48.972994 21877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:48.974277 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:48.975594 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:48.989156 21884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:48.989179 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:48.989225 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:48.989866 21877 server_base.cc:1034] running on GCE node
I20260811 12:34:48.990446 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:48.991421 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:48.992671 21877 hybrid_clock.cc:648] HybridClock initialized: now 1786451688992587 us; error 40 us; skew 500 ppm
I20260811 12:34:48.993006 21877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:48 dist-test-slave-lgr5 krb5kdc[21681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451688, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:48.996265 21877 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:34:48.997795 21877 webserver.cc:460] Webserver started at http://127.3.204.65:42321/ using document root <none> and password file <none>
I20260811 12:34:48.998528 21877 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:48.998658 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:48.998970 21877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:49.002113 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "2361f2d029ae4d5189aa97af6274d49e"
format_stamp: "Formatted at 2026-08-11 12:34:48 on dist-test-slave-lgr5"
server_key: "d4123800ad7b5100539488513e876d4e"
server_key_iv: "f474e2b2280a2040d304de667c2da605"
server_key_version: "encryptionkey@0"
I20260811 12:34:49.002987 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "2361f2d029ae4d5189aa97af6274d49e"
format_stamp: "Formatted at 2026-08-11 12:34:48 on dist-test-slave-lgr5"
server_key: "d4123800ad7b5100539488513e876d4e"
server_key_iv: "f474e2b2280a2040d304de667c2da605"
server_key_version: "encryptionkey@0"
I20260811 12:34:49.008646 21877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260811 12:34:49.012306 21902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:49.013793 21877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:34:49.013974 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "2361f2d029ae4d5189aa97af6274d49e"
format_stamp: "Formatted at 2026-08-11 12:34:48 on dist-test-slave-lgr5"
server_key: "d4123800ad7b5100539488513e876d4e"
server_key_iv: "f474e2b2280a2040d304de667c2da605"
server_key_version: "encryptionkey@0"
I20260811 12:34:49.014165 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:49.027824 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:49.031968 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:49.032279 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:49.033250 21877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:49.034659 21877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:49.034740 21877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:49.034860 21877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:49.034927 21877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:49.068526 21877 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:42463
I20260811 12:34:49.068593 22096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:42463 every 8 connection(s)
I20260811 12:34:49.069818 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:34:49.070344 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 21877
I20260811 12:34:49.070565 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:34:49.071327 3889 external_mini_cluster.cc:1428] Setting key fe38122a87517b2a79bea27b14ad4764
Aug 11 12:34:49 dist-test-slave-lgr5 krb5kdc[21681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451688, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:34:49.092602 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:49.072476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:35923 (local address 127.3.204.126:42729)
0811 12:34:49.072972 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:49.072978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:49.074108 (+ 1130us) server_negotiation.cc:407] Connection header received
0811 12:34:49.075953 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:49.075958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:49.076039 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:49.076165 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:49.078132 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:49.078864 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:49.081053 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:49.081465 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:49.081569 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:49.085702 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:49.085709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:49.085714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:49.085760 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:49.089146 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:49.090031 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:49.090037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:49.090040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:49.090195 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:49.090654 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:49.090659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:49.090662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:49.091026 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:49.091172 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:49.091980 (+ 808us) server_negotiation.cc:299] Negotiation successful
0811 12:34:49.092251 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":232,"threads_started":1}
I20260811 12:34:49.093788 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:49.073104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42729 (local address 127.3.204.65:35923)
0811 12:34:49.073949 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:49.073988 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:49.075666 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:49.076445 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:49.076457 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:49.077151 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:49.077943 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:49.077969 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:49.079025 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:49.079030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:49.080893 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:49.080922 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:49.081697 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:49.081704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:49.081997 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:49.082738 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:49.082768 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:49.085413 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:49.089347 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:49.089358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:49.089378 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:49.089893 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:49.090322 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:49.090325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:49.090327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:49.090540 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:49.091167 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:49.091173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:49.091587 (+ 414us) client_negotiation.cc:769] Sending connection context
0811 12:34:49.091850 (+ 263us) client_negotiation.cc:240] Negotiation successful
0811 12:34:49.092440 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":385,"threads_started":1}
I20260811 12:34:49.095420 22098 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42729
I20260811 12:34:49.095754 22098 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:49.096594 22098 heartbeater.cc:511] Master 127.3.204.126:42729 requested a full tablet report, sending...
I20260811 12:34:49.098867 21765 ts_manager.cc:194] Registered new tserver with Master: 2361f2d029ae4d5189aa97af6274d49e (127.3.204.65:42463)
I20260811 12:34:49.100440 21765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:35923
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:34:49.135459 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42729
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:49.471835 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:49.472170 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:49.472317 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:49.479729 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:49.479882 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:49.480041 22104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:34:49.490331 22104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:49.491791 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:49.493232 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:49.506090 22112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:49.506155 22114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:49.506176 22127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:49.507269 22104 server_base.cc:1034] running on GCE node
I20260811 12:34:49.507912 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:49.508865 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:49.510107 22104 hybrid_clock.cc:648] HybridClock initialized: now 1786451689510051 us; error 41 us; skew 500 ppm
I20260811 12:34:49.510488 22104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:49 dist-test-slave-lgr5 krb5kdc[21681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451689, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:49.513921 22104 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:34:49.515534 22104 webserver.cc:460] Webserver started at http://127.3.204.66:36669/ using document root <none> and password file <none>
I20260811 12:34:49.516393 22104 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:49.516546 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:49.516862 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:49.519958 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "82f3c596716d4e82b4bc3cca7ddb014e"
format_stamp: "Formatted at 2026-08-11 12:34:49 on dist-test-slave-lgr5"
server_key: "b97cdce0d9036ebb2d96098cf5d1e930"
server_key_iv: "40c22351550c4f84d109577fc6048396"
server_key_version: "encryptionkey@0"
I20260811 12:34:49.520771 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "82f3c596716d4e82b4bc3cca7ddb014e"
format_stamp: "Formatted at 2026-08-11 12:34:49 on dist-test-slave-lgr5"
server_key: "b97cdce0d9036ebb2d96098cf5d1e930"
server_key_iv: "40c22351550c4f84d109577fc6048396"
server_key_version: "encryptionkey@0"
I20260811 12:34:49.526185 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:34:49.529971 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:49.531473 22104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:34:49.531664 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "82f3c596716d4e82b4bc3cca7ddb014e"
format_stamp: "Formatted at 2026-08-11 12:34:49 on dist-test-slave-lgr5"
server_key: "b97cdce0d9036ebb2d96098cf5d1e930"
server_key_iv: "40c22351550c4f84d109577fc6048396"
server_key_version: "encryptionkey@0"
I20260811 12:34:49.531864 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:49.547577 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:49.552019 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:49.552338 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:49.553347 22104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:49.554754 22104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:49.554828 22104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:49.554917 22104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:49.554963 22104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:49.587177 22104 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:38641
I20260811 12:34:49.587232 22342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:38641 every 8 connection(s)
I20260811 12:34:49.588601 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:34:49.593065 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 22104
I20260811 12:34:49.593283 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:34:49.593875 3889 external_mini_cluster.cc:1428] Setting key 9356f6caf329449107bc23a6dffbc31a
Aug 11 12:34:49 dist-test-slave-lgr5 krb5kdc[21681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451689, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:34:49.612251 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:49.591134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:52915 (local address 127.3.204.126:42729)
0811 12:34:49.591346 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:49.591353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:49.592669 (+ 1316us) server_negotiation.cc:407] Connection header received
0811 12:34:49.595168 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:49.595174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:49.595252 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:49.595390 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:49.597356 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:49.598059 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:49.600254 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:49.600809 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:49.600923 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:49.605406 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:49.605414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:49.605419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:49.605464 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:49.608424 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:49.609403 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:49.609408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:49.609411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:49.609554 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:49.610136 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:49.610142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:49.610145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:49.610551 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:49.610698 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:49.611587 (+ 889us) server_negotiation.cc:299] Negotiation successful
0811 12:34:49.611880 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":70}
I20260811 12:34:49.613234 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:49.591676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42729 (local address 127.3.204.66:52915)
0811 12:34:49.592500 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:49.592539 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:49.594824 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:49.595581 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:49.595589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:49.596293 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:49.597124 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:49.597151 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:49.598229 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:49.598236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:49.600059 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:49.600075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:49.601049 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:49.601056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:49.601353 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:49.602140 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:49.602169 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:49.605094 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:49.608582 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:49.608592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:49.608612 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:49.609231 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:49.609703 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:49.609707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:49.609710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:49.609958 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:49.610686 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:49.610694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:49.611103 (+ 409us) client_negotiation.cc:769] Sending connection context
0811 12:34:49.611437 (+ 334us) client_negotiation.cc:240] Negotiation successful
0811 12:34:49.611921 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":336,"threads_started":1}
I20260811 12:34:49.614815 22343 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42729
I20260811 12:34:49.615149 22343 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:49.615916 22343 heartbeater.cc:511] Master 127.3.204.126:42729 requested a full tablet report, sending...
I20260811 12:34:49.617475 21765 ts_manager.cc:194] Registered new tserver with Master: 82f3c596716d4e82b4bc3cca7ddb014e (127.3.204.66:38641)
I20260811 12:34:49.618474 21765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:52915
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:34:49.659982 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42729
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:34:49.997594 22351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:34:49.997871 22351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:34:49.998016 22351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:34:50.005409 22351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:34:50.005577 22351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:34:50.005720 22351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:34:50.016047 22351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:42729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:34:50.017383 22351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:34:50.018765 22351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:34:50.033324 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:50.033418 22362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:34:50.033447 22363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:34:50.034215 22351 server_base.cc:1034] running on GCE node
I20260811 12:34:50.034852 22351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:34:50.035941 22351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:34:50.037164 22351 hybrid_clock.cc:648] HybridClock initialized: now 1786451690037085 us; error 72 us; skew 500 ppm
I20260811 12:34:50.037520 22351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:34:50 dist-test-slave-lgr5 krb5kdc[21681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451690, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:34:50.040779 22351 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:34:50.042243 22351 webserver.cc:460] Webserver started at http://127.3.204.67:36787/ using document root <none> and password file <none>
I20260811 12:34:50.042941 22351 fs_manager.cc:359] Metadata directory not provided
I20260811 12:34:50.043042 22351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:34:50.043291 22351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:34:50.046265 22351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "e445720561544b79986e1e719ecd8e21"
format_stamp: "Formatted at 2026-08-11 12:34:50 on dist-test-slave-lgr5"
server_key: "0a4e72ee1365bf46c6d9ace4a7017d80"
server_key_iv: "1a8b0be84d8b7ff4288e77bc3e92dd2c"
server_key_version: "encryptionkey@0"
I20260811 12:34:50.047080 22351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "e445720561544b79986e1e719ecd8e21"
format_stamp: "Formatted at 2026-08-11 12:34:50 on dist-test-slave-lgr5"
server_key: "0a4e72ee1365bf46c6d9ace4a7017d80"
server_key_iv: "1a8b0be84d8b7ff4288e77bc3e92dd2c"
server_key_version: "encryptionkey@0"
I20260811 12:34:50.052345 22351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260811 12:34:50.055843 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:50.057405 22351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:34:50.057575 22351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "e445720561544b79986e1e719ecd8e21"
format_stamp: "Formatted at 2026-08-11 12:34:50 on dist-test-slave-lgr5"
server_key: "0a4e72ee1365bf46c6d9ace4a7017d80"
server_key_iv: "1a8b0be84d8b7ff4288e77bc3e92dd2c"
server_key_version: "encryptionkey@0"
I20260811 12:34:50.057783 22351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:34:50.089471 22351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:34:50.094026 22351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:34:50.094298 22351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:34:50.095323 22351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:34:50.096693 22351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:34:50.096769 22351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:50.096858 22351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:34:50.096909 22351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:34:50.103578 22098 heartbeater.cc:503] Master 127.3.204.126:42729 was elected leader, sending a full tablet report...
I20260811 12:34:50.128127 22351 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:41295
I20260811 12:34:50.128191 22599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:41295 every 8 connection(s)
I20260811 12:34:50.129474 22351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:34:50.138347 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 22351
I20260811 12:34:50.138554 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:34:50.139092 3889 external_mini_cluster.cc:1428] Setting key 206458c4394f956cecf386ce8d2b57aa
Aug 11 12:34:50 dist-test-slave-lgr5 krb5kdc[21681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451690, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:34:50.150261 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.132258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:48039 (local address 127.3.204.126:42729)
0811 12:34:50.132760 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.132764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.133630 (+ 866us) server_negotiation.cc:407] Connection header received
0811 12:34:50.135541 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.135547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.135637 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.135736 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:50.137417 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.138270 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.140064 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.140554 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.140672 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.144287 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:50.144294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:50.144298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:50.144335 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:50.146912 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:50.147687 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:50.147692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:50.147695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:50.147843 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:50.148347 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:50.148350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:50.148353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:50.148637 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:50.148778 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.149571 (+ 793us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.149843 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":241,"threads_started":1}
I20260811 12:34:50.151300 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.132734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:42729 (local address 127.3.204.67:48039)
0811 12:34:50.133467 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:50.133510 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:50.135253 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:50.136014 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:50.136026 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:50.136599 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:34:50.137213 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.137232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.138443 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.138447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.139907 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.139921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.140804 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.140812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.141055 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.141717 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:34:50.141742 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:34:50.144043 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:34:50.147071 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:50.147082 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:50.147099 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:50.147558 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:50.147955 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:34:50.147961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:34:50.147965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:34:50.148227 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:34:50.148780 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:34:50.148794 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:34:50.149179 (+ 385us) client_negotiation.cc:769] Sending connection context
0811 12:34:50.149448 (+ 269us) client_negotiation.cc:240] Negotiation successful
0811 12:34:50.150056 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":292,"threads_started":1}
I20260811 12:34:50.152944 22602 heartbeater.cc:347] Connected to a master server at 127.3.204.126:42729
I20260811 12:34:50.153264 22602 heartbeater.cc:464] Registering TS with master...
I20260811 12:34:50.154072 22602 heartbeater.cc:511] Master 127.3.204.126:42729 requested a full tablet report, sending...
I20260811 12:34:50.155633 21765 ts_manager.cc:194] Registered new tserver with Master: e445720561544b79986e1e719ecd8e21 (127.3.204.67:41295)
I20260811 12:34:50.156625 21765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:48039
I20260811 12:34:50.167166 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:34:50.185595 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.172443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.3.204.126:42729)
0811 12:34:50.172684 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.172692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.172953 (+ 261us) server_negotiation.cc:407] Connection header received
0811 12:34:50.173154 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.173159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.173237 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.173323 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:50.175016 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.175763 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.177180 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.177674 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.177807 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.180225 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:50.180230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:50.180233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:50.180271 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:50.182749 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:50.183440 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:50.183446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:50.183450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:50.183604 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:50.183992 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:50.183998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:50.184000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:50.184406 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:50.184653 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.185032 (+ 379us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.185278 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 12:34:50.189775 21765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:34:50.197008 21765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:34:50.221750 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.209780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:38641 (local address 127.0.0.1:46878)
0811 12:34:50.211105 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:50.211156 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:50.211392 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:50.212707 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:50.212714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:50.212746 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:50.213233 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.213247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.215649 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.215655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.218376 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.218394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.220485 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.220493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.220694 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.220762 (+ 68us) client_negotiation.cc:769] Sending connection context
0811 12:34:50.220892 (+ 130us) client_negotiation.cc:240] Negotiation successful
0811 12:34:50.221218 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":4,"thread_start_us":2763,"threads_started":1}
I20260811 12:34:50.221848 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.210327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:41295 (local address 127.0.0.1:45394)
0811 12:34:50.212822 (+ 2495us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:50.212933 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:50.213077 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:50.213808 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:50.213813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:50.213839 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:50.214266 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.214279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.216284 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.216289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.218665 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.218680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.220943 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.220951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.221161 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.221195 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 12:34:50.221285 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 12:34:50.221444 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2361,"thread_start_us":774,"threads_started":1}
I20260811 12:34:50.222937 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.210924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.3.204.67:41295)
0811 12:34:50.212419 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.212427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.213213 (+ 786us) server_negotiation.cc:407] Connection header received
0811 12:34:50.213323 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.213331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.213608 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.213798 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:50.214505 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.216075 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.218864 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.220369 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.220515 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.220786 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.222393 (+ 1607us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.222612 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":1165,"threads_started":1}
I20260811 12:34:50.223476 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.209902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.3.204.66:38641)
0811 12:34:50.211981 (+ 2079us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.211992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.212025 (+ 33us) server_negotiation.cc:407] Connection header received
0811 12:34:50.212155 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.212163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.212430 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.213467 (+ 1037us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:50.213701 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.215495 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.218597 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.220265 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.222510 (+ 2245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.222731 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.222817 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.223182 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1912,"thread_start_us":1808,"threads_started":1}
I20260811 12:34:50.227823 22474 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2dbcc34cf3451e84f7b9e3bd143440 (DEFAULT_TABLE table=test-table [id=941a68b9e1df4ebf90d1b21bfca9b705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:50.227895 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2dbcc34cf3451e84f7b9e3bd143440 (DEFAULT_TABLE table=test-table [id=941a68b9e1df4ebf90d1b21bfca9b705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:50.229696 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2dbcc34cf3451e84f7b9e3bd143440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:50.229709 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2dbcc34cf3451e84f7b9e3bd143440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:50.230306 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.213734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:42463 (local address 127.0.0.1:55318)
0811 12:34:50.214538 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:50.214564 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:50.214681 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:50.215805 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:50.215810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:50.215835 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:50.216389 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.216405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.222735 (+ 6330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.222740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.224817 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.224835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.229460 (+ 4625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.229467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.229666 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.229701 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:34:50.229799 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 12:34:50.229950 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":521,"threads_started":1}
I20260811 12:34:50.231072 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.213160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.3.204.65:42463)
0811 12:34:50.215283 (+ 2123us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.215290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.215314 (+ 24us) server_negotiation.cc:407] Connection header received
0811 12:34:50.215400 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.215406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.215588 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.215809 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:50.216636 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.220218 (+ 3582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.225043 (+ 4825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.228887 (+ 3844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.229043 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.229307 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.230572 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.230762 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1971,"thread_start_us":1817,"threads_started":1}
I20260811 12:34:50.235174 21986 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2dbcc34cf3451e84f7b9e3bd143440 (DEFAULT_TABLE table=test-table [id=941a68b9e1df4ebf90d1b21bfca9b705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:34:50.236829 21986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2dbcc34cf3451e84f7b9e3bd143440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:34:50.245317 22635 tablet_bootstrap.cc:492] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Bootstrap starting.
I20260811 12:34:50.249029 22635 tablet_bootstrap.cc:654] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:50.250684 22635 log.cc:824] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:50.254074 22638 tablet_bootstrap.cc:492] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Bootstrap starting.
I20260811 12:34:50.254088 22642 tablet_bootstrap.cc:492] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Bootstrap starting.
I20260811 12:34:50.257939 22638 tablet_bootstrap.cc:654] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:50.257956 22642 tablet_bootstrap.cc:654] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Neither blocks nor log segments found. Creating new log.
I20260811 12:34:50.259299 22642 log.cc:824] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:50.259291 22638 log.cc:824] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Log is configured to *not* fsync() on all Append() calls
I20260811 12:34:50.259931 22635 tablet_bootstrap.cc:492] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: No bootstrap required, opened a new log
I20260811 12:34:50.260291 22635 ts_tablet_manager.cc:1397] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260811 12:34:50.271267 22642 tablet_bootstrap.cc:492] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: No bootstrap required, opened a new log
I20260811 12:34:50.270990 22635 raft_consensus.cc:348] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.271517 22635 raft_consensus.cc:374] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:50.271611 22642 ts_tablet_manager.cc:1397] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.009s
I20260811 12:34:50.271634 22635 raft_consensus.cc:729] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e445720561544b79986e1e719ecd8e21, State: Initialized, Role: FOLLOWER
I20260811 12:34:50.272444 22638 tablet_bootstrap.cc:492] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: No bootstrap required, opened a new log
I20260811 12:34:50.272369 22635 consensus_queue.cc:260] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.272732 22638 ts_tablet_manager.cc:1397] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20260811 12:34:50.273634 22602 heartbeater.cc:503] Master 127.3.204.126:42729 was elected leader, sending a full tablet report...
I20260811 12:34:50.274259 22635 ts_tablet_manager.cc:1428] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260811 12:34:50.282254 22642 raft_consensus.cc:348] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.282842 22642 raft_consensus.cc:374] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:50.282967 22642 raft_consensus.cc:729] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2361f2d029ae4d5189aa97af6274d49e, State: Initialized, Role: FOLLOWER
I20260811 12:34:50.283181 22638 raft_consensus.cc:348] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.283631 22638 raft_consensus.cc:374] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:34:50.283739 22638 raft_consensus.cc:729] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f3c596716d4e82b4bc3cca7ddb014e, State: Initialized, Role: FOLLOWER
I20260811 12:34:50.283689 22642 consensus_queue.cc:260] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.284327 22638 consensus_queue.cc:260] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.285804 22642 ts_tablet_manager.cc:1428] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260811 12:34:50.286118 22638 ts_tablet_manager.cc:1428] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260811 12:34:50.286412 22343 heartbeater.cc:503] Master 127.3.204.126:42729 was elected leader, sending a full tablet report...
W20260811 12:34:50.322356 22099 tablet.cc:2369] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:34:50.340646 22345 tablet.cc:2369] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:34:50.381435 22606 tablet.cc:2369] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:34:50.662741 22647 raft_consensus.cc:482] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:34:50.663115 22647 raft_consensus.cc:504] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.665027 22647 leader_election.cc:288] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f3c596716d4e82b4bc3cca7ddb014e (127.3.204.66:38641), e445720561544b79986e1e719ecd8e21 (127.3.204.67:41295)
I20260811 12:34:50.672001 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.665500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:45511 (local address 127.3.204.67:41295)
0811 12:34:50.665693 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.665699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.666340 (+ 641us) server_negotiation.cc:407] Connection header received
0811 12:34:50.666562 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.666566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.666629 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.666704 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:50.667286 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.667996 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.669934 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.671061 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.671213 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.671271 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.671635 (+ 364us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.671730 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:34:50.672223 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.665462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:41295 (local address 127.3.204.65:45511)
0811 12:34:50.666217 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:50.666234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:50.666409 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:50.666743 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:50.666748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:50.666786 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:50.667135 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.667145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.668149 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.668153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.669761 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.669782 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.671288 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.671295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.671474 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.671511 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 12:34:50.671614 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 12:34:50.671746 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":21,"thread_start_us":433,"threads_started":1}
I20260811 12:34:50.672854 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.665625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:50489 (local address 127.3.204.66:38641)
0811 12:34:50.665825 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.665832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.666763 (+ 931us) server_negotiation.cc:407] Connection header received
0811 12:34:50.667005 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.667010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.667073 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.667149 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:34:50.667669 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.668463 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.670604 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.671736 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.671892 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.671965 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.672449 (+ 484us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.672592 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 12:34:50.672992 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.665595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:38641 (local address 127.3.204.65:50489)
0811 12:34:50.666644 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0811 12:34:50.666659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:34:50.666851 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:34:50.667189 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:34:50.667193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:34:50.667217 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:34:50.667508 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.667517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.668605 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.668609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.670408 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:34:50.670424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.671996 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.672003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:34:50.672204 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.672239 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:34:50.672324 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 12:34:50.672626 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":19,"spinlock_wait_cycles":26880,"thread_start_us":741,"threads_started":1}
I20260811 12:34:50.673224 22519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2dbcc34cf3451e84f7b9e3bd143440" candidate_uuid: "2361f2d029ae4d5189aa97af6274d49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e445720561544b79986e1e719ecd8e21" is_pre_election: true
I20260811 12:34:50.673703 22519 raft_consensus.cc:2454] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2361f2d029ae4d5189aa97af6274d49e in term 0.
I20260811 12:34:50.673964 22260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2dbcc34cf3451e84f7b9e3bd143440" candidate_uuid: "2361f2d029ae4d5189aa97af6274d49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" is_pre_election: true
I20260811 12:34:50.674360 21908 leader_election.cc:302] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2361f2d029ae4d5189aa97af6274d49e, e445720561544b79986e1e719ecd8e21; no voters:
I20260811 12:34:50.674549 22260 raft_consensus.cc:2454] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2361f2d029ae4d5189aa97af6274d49e in term 0.
I20260811 12:34:50.674853 22647 raft_consensus.cc:2781] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:34:50.675019 22647 raft_consensus.cc:482] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:34:50.675117 22647 raft_consensus.cc:3037] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:50.677433 22647 raft_consensus.cc:504] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.678020 22647 leader_election.cc:288] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [CANDIDATE]: Term 1 election: Requested vote from peers 82f3c596716d4e82b4bc3cca7ddb014e (127.3.204.66:38641), e445720561544b79986e1e719ecd8e21 (127.3.204.67:41295)
I20260811 12:34:50.678491 22519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2dbcc34cf3451e84f7b9e3bd143440" candidate_uuid: "2361f2d029ae4d5189aa97af6274d49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e445720561544b79986e1e719ecd8e21"
I20260811 12:34:50.678499 22260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2dbcc34cf3451e84f7b9e3bd143440" candidate_uuid: "2361f2d029ae4d5189aa97af6274d49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f3c596716d4e82b4bc3cca7ddb014e"
I20260811 12:34:50.678679 22260 raft_consensus.cc:3037] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:50.678683 22519 raft_consensus.cc:3037] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:34:50.680917 22519 raft_consensus.cc:2454] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2361f2d029ae4d5189aa97af6274d49e in term 1.
I20260811 12:34:50.681003 22260 raft_consensus.cc:2454] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2361f2d029ae4d5189aa97af6274d49e in term 1.
I20260811 12:34:50.681342 21908 leader_election.cc:302] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2361f2d029ae4d5189aa97af6274d49e, e445720561544b79986e1e719ecd8e21; no voters:
I20260811 12:34:50.681686 22647 raft_consensus.cc:2781] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:34:50.682163 22647 raft_consensus.cc:686] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 1 LEADER]: Becoming Leader. State: Replica: 2361f2d029ae4d5189aa97af6274d49e, State: Running, Role: LEADER
I20260811 12:34:50.682729 22647 consensus_queue.cc:237] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } }
I20260811 12:34:50.687395 21765 catalog_manager.cc:5495] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2361f2d029ae4d5189aa97af6274d49e (127.3.204.65). New cstate: current_term: 1 leader_uuid: "2361f2d029ae4d5189aa97af6274d49e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2361f2d029ae4d5189aa97af6274d49e" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 42463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 } health_report { overall_health: UNKNOWN } } }
I20260811 12:34:50.793807 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.788204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.3.204.65:42463)
0811 12:34:50.788641 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.788646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.788844 (+ 198us) server_negotiation.cc:407] Connection header received
0811 12:34:50.789051 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.789055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.789137 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.789285 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:34:50.789675 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.790368 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.791712 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.792178 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.792281 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.792679 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:34:50.793298 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:34:50.793406 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.793497 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.793574 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260811 12:34:50.801973 22260 raft_consensus.cc:1261] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 1 FOLLOWER]: Refusing update from remote peer 2361f2d029ae4d5189aa97af6274d49e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:50.801972 22519 raft_consensus.cc:1261] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 1 FOLLOWER]: Refusing update from remote peer 2361f2d029ae4d5189aa97af6274d49e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:34:50.802919 22655 consensus_queue.cc:1035] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f3c596716d4e82b4bc3cca7ddb014e" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:50.803294 22647 consensus_queue.cc:1035] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e445720561544b79986e1e719ecd8e21" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:34:50.836956 21764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:50.837234 21764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:34:50.840806 21764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c14535dd6ff4e289afde6e4972da168: Sending DeleteTablet for 3 replicas of tablet 7f2dbcc34cf3451e84f7b9e3bd143440
I20260811 12:34:50.842109 22474 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2dbcc34cf3451e84f7b9e3bd143440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:50 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:45394
I20260811 12:34:50.842133 21986 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2dbcc34cf3451e84f7b9e3bd143440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:50 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:55318
I20260811 12:34:50.843111 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2dbcc34cf3451e84f7b9e3bd143440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:34:50 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:46878
I20260811 12:34:50.844029 22687 tablet_replica.cc:330] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: stopping tablet replica
I20260811 12:34:50.844225 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 21877
I20260811 12:34:50.844543 22687 raft_consensus.cc:2229] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:50.845080 22687 raft_consensus.cc:2258] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:50.847012 22687 ts_tablet_manager.cc:1905] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:50.853201 22685 tablet_replica.cc:330] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: stopping tablet replica
I20260811 12:34:50.853674 22685 raft_consensus.cc:2229] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:34:50.854410 22685 raft_consensus.cc:2258] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:50.854645 22687 ts_tablet_manager.cc:1918] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:50.854828 22687 log.cc:1196] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/7f2dbcc34cf3451e84f7b9e3bd143440
I20260811 12:34:50.855409 22687 ts_tablet_manager.cc:1939] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 82f3c596716d4e82b4bc3cca7ddb014e: Deleting consensus metadata
I20260811 12:34:50.857547 21744 catalog_manager.cc:4841] TS 82f3c596716d4e82b4bc3cca7ddb014e (127.3.204.66:38641): tablet 7f2dbcc34cf3451e84f7b9e3bd143440 (table test-table [id=941a68b9e1df4ebf90d1b21bfca9b705]) successfully deleted
I20260811 12:34:50.858498 22685 ts_tablet_manager.cc:1905] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:50.865079 22684 tablet_replica.cc:330] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: stopping tablet replica
I20260811 12:34:50.865563 22684 raft_consensus.cc:2229] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:34:50.865648 22685 ts_tablet_manager.cc:1918] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:50.865824 22685 log.cc:1196] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/7f2dbcc34cf3451e84f7b9e3bd143440
I20260811 12:34:50.866147 22684 raft_consensus.cc:2258] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:34:50.866451 22685 ts_tablet_manager.cc:1939] T 7f2dbcc34cf3451e84f7b9e3bd143440 P 2361f2d029ae4d5189aa97af6274d49e: Deleting consensus metadata
I20260811 12:34:50.868391 22684 ts_tablet_manager.cc:1905] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:34:50.868701 21744 catalog_manager.cc:4841] TS 2361f2d029ae4d5189aa97af6274d49e (127.3.204.65:42463): tablet 7f2dbcc34cf3451e84f7b9e3bd143440 (table test-table [id=941a68b9e1df4ebf90d1b21bfca9b705]) successfully deleted
Aug 11 12:34:50 dist-test-slave-lgr5 krb5kdc[21681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:34:50.874586 22684 ts_tablet_manager.cc:1918] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:34:50.874758 22684 log.cc:1196] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/7f2dbcc34cf3451e84f7b9e3bd143440
I20260811 12:34:50.875294 22684 ts_tablet_manager.cc:1939] T 7f2dbcc34cf3451e84f7b9e3bd143440 P e445720561544b79986e1e719ecd8e21: Deleting consensus metadata
I20260811 12:34:50.877095 21743 catalog_manager.cc:4841] TS e445720561544b79986e1e719ecd8e21 (127.3.204.67:41295): tablet 7f2dbcc34cf3451e84f7b9e3bd143440 (table test-table [id=941a68b9e1df4ebf90d1b21bfca9b705]) successfully deleted
I20260811 12:34:50.879210 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:50.847155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.3.204.65:42463)
0811 12:34:50.847345 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:50.847352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:50.859601 (+ 12249us) server_negotiation.cc:407] Connection header received
0811 12:34:50.859804 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:50.859813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:50.859896 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:50.860017 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:50.861743 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.863255 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.865355 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:50.866000 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:50.866152 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:50.871781 (+ 5629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:50.871798 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:50.871818 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:50.871862 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:50.876076 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:50.876996 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:50.877006 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:50.877010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:50.877204 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:50.877696 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:50.877700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:50.877702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:50.878103 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:50.878337 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:50.878705 (+ 368us) server_negotiation.cc:299] Negotiation successful
0811 12:34:50.878891 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":65}
I20260811 12:34:50.879952 22084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:50.979385 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 21877
I20260811 12:34:51.007503 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 22104
Aug 11 12:34:51 dist-test-slave-lgr5 krb5kdc[21681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:34:51.022161 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:51.008356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.3.204.66:38641)
0811 12:34:51.008583 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:51.008588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:51.008607 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:34:51.008739 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:51.008743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:51.008806 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:51.008893 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:51.010101 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:51.010912 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:51.012234 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:51.012667 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:51.012768 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:51.016835 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:51.016852 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:51.016869 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:51.016911 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:51.019697 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:51.020363 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:51.020371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:51.020374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:51.020507 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:51.020821 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:51.020823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:51.020825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:51.021212 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:51.021466 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:51.021653 (+ 187us) server_negotiation.cc:299] Negotiation successful
0811 12:34:51.021835 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260811 12:34:51.022871 22336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:51.112380 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 22104
I20260811 12:34:51.138736 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 22351
Aug 11 12:34:51 dist-test-slave-lgr5 krb5kdc[21681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:34:51.154700 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:34:51.139694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.3.204.67:41295)
0811 12:34:51.139887 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 12:34:51.139894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:34:51.139917 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:34:51.139980 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:34:51.139985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:34:51.140059 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:34:51.140190 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:34:51.141426 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:51.142592 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:51.143830 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:34:51.144502 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:34:51.144617 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:34:51.149061 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:34:51.149075 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:34:51.149092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:34:51.149135 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:34:51.151862 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:51.152606 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:51.152614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:51.152617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:51.152755 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:34:51.153118 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:34:51.153121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:34:51.153123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:34:51.153522 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:34:51.153805 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0811 12:34:51.154126 (+ 321us) server_negotiation.cc:299] Negotiation successful
0811 12:34:51.154365 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:34:51.155501 22591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:51.249516 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 22351
I20260811 12:34:51.266103 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 21708
I20260811 12:34:51.267392 21816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:34:51.377776 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 21708
2026-08-11T12:34:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4924 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:34:51 dist-test-slave-lgr5 krb5kdc[22700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:34:51 dist-test-slave-lgr5 krb5kdc[22700](info): set up 2 sockets
Aug 11 12:34:51 dist-test-slave-lgr5 krb5kdc[22700](info): commencing operation
krb5kdc: starting...
W20260811 12:34:52.661892 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:34:52 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:34:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/rangeradmin_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/rangeradmin_127.3.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/rangerlookup_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/rangerlookup_127.3.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/HTTP_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/HTTP_127.3.204.84@KRBTEST.COM.keytab.
I20260811 12:34:53.068325 3889 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1nKHSZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/postgres -l logfile start
2026-08-11 12:34:55.684 UTC [22754] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 12:34:55.684 UTC [22754] LOG: listening on IPv4 address "127.3.204.84", port 37645
2026-08-11 12:34:55.688 UTC [22754] LOG: listening on Unix socket "/tmp/.s.PGSQL.37645"
2026-08-11 12:34:55.711 UTC [22761] LOG: database system was shut down at 2026-08-11 12:34:54 UTC
2026-08-11 12:34:55.715 UTC [22754] LOG: database system is ready to accept connections
I20260811 12:34:56.840696 3889 mini_postgres.cc:96] Postgres bound to 37645
2026-08-11 12:34:56.863 UTC [22777] FATAL: database "slave" does not exist
127.3.204.84:37645 - accepting connections
I20260811 12:34:56.864120 3889 mini_ranger.cc:161] Starting Ranger...
I20260811 12:34:56.895572 3889 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 12:34:57.413770 3889 mini_ranger.cc:87] Created ranger Postgres database
I20260811 12:34:57.413890 3889 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-admin
2026-08-11 12:34:57,721 [I] DB FLAVOR :POSTGRES
2026-08-11 12:34:57,722 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 12:34:57,722 [I] Checking connection..
2026-08-11 12:34:57,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 12:34:57,991 [I] Checking connection passed.
2026-08-11 12:34:57,992 [I] --------- Verifying version history table ---------
2026-08-11 12:34:57,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 12:34:58,233 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 12:34:58,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 12:34:58,503 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 12:34:58,504 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 12:34:58,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 12:34:58.765 UTC [22896] WARNING: there is no transaction in progress
2026-08-11 12:34:58,775 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 12:34:58,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 12:34:59,013 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 12:34:59,014 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 12:34:59,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 12:34:59,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 12:34:59,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lgr5.c.gcp-upstream.internal','N') ;"
2026-08-11 12:34:59,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 12:35:00,051 [I] Table x_portal_user does not exist in database ranger
2026-08-11 12:35:00,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 12:35:00,332 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 12:35:00,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 12:35:00,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 12:35:00,839 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 12:35:00,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 12:35:01.839 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:01.857 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:01.874 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:01.889 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:01.906 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:01.924 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:01.984 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:01.991 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02.006 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02.016 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02.027 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02.041 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02.052 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02.059 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02.066 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02.073 UTC [23105] WARNING: there is no transaction in progress
2026-08-11 12:35:02,739 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 12:35:02,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 12:35:02,996 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 12:35:02,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 12:35:03,255 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 12:35:03,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 12:35:03,517 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 12:35:03,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 12:35:03,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 12:35:04,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lgr5.c.gcp-upstream.internal';"
2026-08-11 12:35:04,292 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 12:35:04,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 12:35:04,552 [I] DB_PATCHES have already been applied
I20260811 12:35:04.558674 3889 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-admin:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1nKHSZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 12:35:04.558772 3889 mini_ranger.cc:193] Using host: 127.3.204.84
I20260811 12:35:04.578248 3889 mini_ranger.cc:227] Ranger admin URL: http://127.3.204.84:46649
Aug 11, 2026 12:35:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 12:35:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 12:35:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 12:35:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 12:35:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 12:35:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 12:35:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 12:35:05.896100 3889 mini_ranger.cc:160] Time spent starting Ranger: real 9.032s user 0.000s sys 0.064s
Aug 11, 2026 12:35:25 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 12:35:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 11, 2026 12:35:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 12:35:25 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 12:35:26 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 12:35:26.863781 3889 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/rangerkms_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/rangerkms_127.3.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.204.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.204.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.204.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/HTTP_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.204.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/HTTP_127.3.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 12:35:27 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451727, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 12:35:27.081323 3889 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 12:35:27.111464 3889 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 12:35:28.019186 3889 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 12:35:28.019330 3889 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms
I20260811 12:35:28.019467 3889 mini_ranger_kms.cc:227] Using postgres at 127.3.204.84:37645
2026-08-11 12:35:28,926 [I] DB FLAVOR :POSTGRES
2026-08-11 12:35:28,927 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 12:35:28,927 [I] Checking connection
2026-08-11 12:35:28,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 12:35:29,179 [I] connection success
2026-08-11 12:35:29,180 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 12:35:29,180 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 12:35:29,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 12:35:29,437 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 12:35:29,437 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 12:35:29,437 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 12:35:29,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:37645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 12:35:29,735 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 12:35:30.255939 3889 mini_ranger_kms.cc:310] Created kms service
I20260811 12:35:30.465652 3889 mini_ranger_kms.cc:326] Created kudu user
I20260811 12:35:30.509178 3889 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 12:35:30 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451727, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:35:30.928153 3889 mini_ranger_kms.cc:384] Added ranger policy
I20260811 12:35:30.932056 3889 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms:/tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 12:35:30.932174 3889 mini_ranger_kms.cc:240] Using host: 127.3.204.84
I20260811 12:35:30.957520 3889 mini_ranger_kms.cc:276] Ranger KMS PID: 23478
I20260811 12:35:30.957729 3889 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.204.84:59051
Aug 11, 2026 12:35:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 12:35:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 12:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 12:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 12:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 12:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 12:35:33 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786451733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.204.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:35:33.577756 3889 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.496s user 0.163s sys 0.419s
I20260811 12:35:33.578164 3889 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 12:35:33.578397 3889 mini_ranger_kms.cc:401] 127.3.204.84:59051/kms/v1/keys
Aug 11 12:35:33 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.204.84@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
Aug 11 12:35:34 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.204.84@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
Aug 11 12:35:36 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:35:36.927652 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37709
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:37709
--ranger_config_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:59051/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:35:37.581348 23574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:35:37.581782 23574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:35:37.582005 23574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:35:37.592099 23574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:35:37.592283 23574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:35:37.592444 23574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:35:37.592608 23574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:35:37.592716 23574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:35:37.606886 23574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.204.84:59051/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:37709
--ranger_config_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:35:37.608569 23574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:35:37.610152 23574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:35:37.625170 23582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:35:37.625252 23583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:35:37.625579 23588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:35:37.626368 23574 server_base.cc:1034] running on GCE node
I20260811 12:35:37.627501 23574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:35:37.629338 23574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:35:37.630595 23574 hybrid_clock.cc:648] HybridClock initialized: now 1786451737630514 us; error 35 us; skew 500 ppm
I20260811 12:35:37.631091 23574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:35:37 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:35:37.635766 23574 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:35:37.637950 23574 webserver.cc:460] Webserver started at http://127.3.204.126:39157/ using document root <none> and password file <none>
I20260811 12:35:37.638720 23574 fs_manager.cc:359] Metadata directory not provided
I20260811 12:35:37.638928 23574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:35:37.639335 23574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 12:35:37 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:35:37.696329 23574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "cd75c58bf8dd4502b0bbe94fbf37e019"
format_stamp: "Formatted at 2026-08-11 12:35:37 on dist-test-slave-lgr5"
server_key: "1907ebd16ecc14bc75a77ca37395023e"
server_key_iv: "9b418f6c0515a765787d0f8d19608fc3"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:37.697446 23574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "cd75c58bf8dd4502b0bbe94fbf37e019"
format_stamp: "Formatted at 2026-08-11 12:35:37 on dist-test-slave-lgr5"
server_key: "1907ebd16ecc14bc75a77ca37395023e"
server_key_iv: "9b418f6c0515a765787d0f8d19608fc3"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:37.704608 23574 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260811 12:35:37.727751 23597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:37.729485 23574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260811 12:35:37.729727 23574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "cd75c58bf8dd4502b0bbe94fbf37e019"
format_stamp: "Formatted at 2026-08-11 12:35:37 on dist-test-slave-lgr5"
server_key: "1907ebd16ecc14bc75a77ca37395023e"
server_key_iv: "9b418f6c0515a765787d0f8d19608fc3"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:37.730037 23574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:35:37.750486 23574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:35:37.756115 23574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:35:37.756568 23574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:35:37.799050 23574 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:37709
I20260811 12:35:37.799185 23699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:37709 every 8 connection(s)
I20260811 12:35:37.800935 23574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:35:37.806468 23702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:35:37.808710 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 23574
I20260811 12:35:37.808933 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:35:37.809702 3889 external_mini_cluster.cc:1428] Setting key 332dc1fb44e63e965f8d568959bf2814
I20260811 12:35:37.827224 23702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019: Bootstrap starting.
Aug 11 12:35:37 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:35:37.832858 23702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019: Neither blocks nor log segments found. Creating new log.
I20260811 12:35:37.834894 23702 log.cc:824] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019: Log is configured to *not* fsync() on all Append() calls
I20260811 12:35:37.839553 23702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019: No bootstrap required, opened a new log
I20260811 12:35:37.840224 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:37.812311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.3.204.126:37709)
0811 12:35:37.813803 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:37.813824 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:37.813907 (+ 83us) server_negotiation.cc:407] Connection header received
0811 12:35:37.816862 (+ 2955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:37.816904 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:37.817552 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:37.818132 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:35:37.819766 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:37.822547 (+ 2781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:37.824036 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:37.825510 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:37.825717 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:37.830298 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:35:37.830310 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:35:37.830327 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:35:37.830409 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:35:37.834970 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:37.835830 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:37.835838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:37.835845 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:37.836159 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:37.836605 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:37.836611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:37.836614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:37.837422 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:35:37.837791 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:37.838337 (+ 546us) server_negotiation.cc:299] Negotiation successful
0811 12:35:37.838752 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":827,"threads_started":1}
I20260811 12:35:37.852008 23702 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd75c58bf8dd4502b0bbe94fbf37e019" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37709 } }
I20260811 12:35:37.852651 23702 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:35:37.852813 23702 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd75c58bf8dd4502b0bbe94fbf37e019, State: Initialized, Role: FOLLOWER
I20260811 12:35:37.853515 23702 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd75c58bf8dd4502b0bbe94fbf37e019" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37709 } }
I20260811 12:35:37.853861 23702 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:35:37.854099 23702 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:35:37.854362 23702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:35:37.858076 23702 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd75c58bf8dd4502b0bbe94fbf37e019" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37709 } }
I20260811 12:35:37.858793 23702 leader_election.cc:302] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd75c58bf8dd4502b0bbe94fbf37e019; no voters:
I20260811 12:35:37.859733 23702 leader_election.cc:288] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:35:37.859937 23712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:35:37.861579 23712 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [term 1 LEADER]: Becoming Leader. State: Replica: cd75c58bf8dd4502b0bbe94fbf37e019, State: Running, Role: LEADER
I20260811 12:35:37.862340 23712 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd75c58bf8dd4502b0bbe94fbf37e019" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37709 } }
I20260811 12:35:37.863083 23702 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:35:37.866422 23715 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd75c58bf8dd4502b0bbe94fbf37e019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd75c58bf8dd4502b0bbe94fbf37e019" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37709 } } }
I20260811 12:35:37.867198 23715 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [sys.catalog]: This master's current role is: LEADER
I20260811 12:35:37.866561 23716 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd75c58bf8dd4502b0bbe94fbf37e019. Latest consensus state: current_term: 1 leader_uuid: "cd75c58bf8dd4502b0bbe94fbf37e019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd75c58bf8dd4502b0bbe94fbf37e019" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37709 } } }
I20260811 12:35:37.868120 23716 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019 [sys.catalog]: This master's current role is: LEADER
I20260811 12:35:37.871269 23723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:35:37.880813 23723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:35:37.886755 23702 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 12:35:37.909221 23723 catalog_manager.cc:1347] Generated new cluster ID: eda34c49a0c24ca89a6a36471111d149
I20260811 12:35:37.909446 23723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:35:37.941471 23723 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:35:37.943821 23723 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:35:37.978323 23723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019: Generated new TSK 0
I20260811 12:35:37.979374 23723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:35:38.840221 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:59051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37709
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:35:39 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786451739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 12:35:39.527415 23799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:35:39.527814 23799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:35:39.528051 23799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:35:39.541433 23799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:35:39.541810 23799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:35:39.542157 23799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:35:39.563122 23799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.204.84:59051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:35:39.565205 23799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:35:39.567453 23799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:35:39.604382 23821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:35:39.605470 23815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:35:39.607678 23823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:35:39.608144 23799 server_base.cc:1034] running on GCE node
I20260811 12:35:39.608868 23799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:35:39.610680 23799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:35:39.611992 23799 hybrid_clock.cc:648] HybridClock initialized: now 1786451739611886 us; error 99 us; skew 500 ppm
I20260811 12:35:39.612525 23799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:35:39 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:35:39.617117 23799 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:35:39.630308 23799 webserver.cc:460] Webserver started at http://127.3.204.65:34647/ using document root <none> and password file <none>
I20260811 12:35:39.631659 23799 fs_manager.cc:359] Metadata directory not provided
I20260811 12:35:39.631910 23799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:35:39.632416 23799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 12:35:39 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:35:39.661214 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "881f5b25bcae4952a9c1358aecc3796a"
format_stamp: "Formatted at 2026-08-11 12:35:39 on dist-test-slave-lgr5"
server_key: "24b825e05a220d2233ba4bd35099344c"
server_key_iv: "f48710ebd3558c0fed898a2ffcf2b2ef"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:39.662761 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "881f5b25bcae4952a9c1358aecc3796a"
format_stamp: "Formatted at 2026-08-11 12:35:39 on dist-test-slave-lgr5"
server_key: "24b825e05a220d2233ba4bd35099344c"
server_key_iv: "f48710ebd3558c0fed898a2ffcf2b2ef"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:39.673554 23799 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.002s
I20260811 12:35:39.721413 23835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:39.723901 23799 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.008s sys 0.012s
I20260811 12:35:39.724217 23799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "881f5b25bcae4952a9c1358aecc3796a"
format_stamp: "Formatted at 2026-08-11 12:35:39 on dist-test-slave-lgr5"
server_key: "24b825e05a220d2233ba4bd35099344c"
server_key_iv: "f48710ebd3558c0fed898a2ffcf2b2ef"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:39.724582 23799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 11 12:35:39 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:35:39.773216 23799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:35:39.781013 23799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:35:39.781538 23799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:35:39.785176 23799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:35:39.788101 23799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:35:39.788239 23799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 12:35:39.788409 23799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:35:39.788551 23799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:40.025534 23799 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:38467
I20260811 12:35:40.025579 24039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:38467 every 8 connection(s)
I20260811 12:35:40.027454 23799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:35:40.028378 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 23799
I20260811 12:35:40.028635 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:35:40.029716 3889 external_mini_cluster.cc:1428] Setting key 0e920fca70082708199061f97ab31e66
Aug 11 12:35:40 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
I20260811 12:35:40.103916 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:40.055259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37709 (local address 127.3.204.65:58927)
0811 12:35:40.061094 (+ 5835us) negotiation.cc:107] Waiting for socket to connect
0811 12:35:40.061161 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0811 12:35:40.065365 (+ 4204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:35:40.074245 (+ 8880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:35:40.074261 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:35:40.075324 (+ 1063us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:35:40.076827 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:40.076867 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:40.078911 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:40.078917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:40.081967 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:40.082020 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:40.083953 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:40.083964 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:40.084434 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:40.085571 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:35:40.085606 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:35:40.089946 (+ 4340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:35:40.098120 (+ 8174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:35:40.098134 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:35:40.098157 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:35:40.098954 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:35:40.099555 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:35:40.099561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:35:40.099564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:35:40.099884 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:35:40.100741 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:35:40.100750 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:35:40.101292 (+ 542us) client_negotiation.cc:769] Sending connection context
0811 12:35:40.101668 (+ 376us) client_negotiation.cc:240] Negotiation successful
0811 12:35:40.102313 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5373,"thread_start_us":4935,"threads_started":1}
I20260811 12:35:40.105034 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:40.056245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:58927 (local address 127.3.204.126:37709)
0811 12:35:40.073170 (+ 16925us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:40.073177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:40.073210 (+ 33us) server_negotiation.cc:407] Connection header received
0811 12:35:40.073293 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:40.073299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:40.073401 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:40.073573 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:35:40.077145 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:40.078622 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:40.082230 (+ 3608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:40.083249 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:40.083374 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:40.090493 (+ 7119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:35:40.090504 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:35:40.090510 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:35:40.090600 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:35:40.097811 (+ 7211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:40.099165 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:40.099171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:40.099175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:40.099377 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:40.100053 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:40.100059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:40.100062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:40.100485 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:35:40.100636 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:40.104323 (+ 3687us) server_negotiation.cc:299] Negotiation successful
0811 12:35:40.104631 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16735,"thread_start_us":16560,"threads_started":1}
I20260811 12:35:40.107342 24040 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37709
I20260811 12:35:40.107718 24040 heartbeater.cc:464] Registering TS with master...
I20260811 12:35:40.108882 24040 heartbeater.cc:511] Master 127.3.204.126:37709 requested a full tablet report, sending...
I20260811 12:35:40.112180 23622 ts_manager.cc:194] Registered new tserver with Master: 881f5b25bcae4952a9c1358aecc3796a (127.3.204.65:38467)
I20260811 12:35:40.114856 23622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:58927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
W20260811 12:35:40.138989 3889 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.204.66: real 0.105s user 0.002s sys 0.026s
I20260811 12:35:40.140026 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:59051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37709
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:35:40 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
W20260811 12:35:40.907117 24051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:35:40.907517 24051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:35:40.907737 24051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:35:40.918007 24051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:35:40.918231 24051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:35:40.918505 24051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:35:40.932402 24051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.204.84:59051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:35:40.933935 24051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:35:40.935412 24051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:35:40.954468 24076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:35:40.954675 24073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:35:40.955120 24069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:35:40.957013 24051 server_base.cc:1034] running on GCE node
I20260811 12:35:40.957731 24051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:35:40.958801 24051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:35:40.960011 24051 hybrid_clock.cc:648] HybridClock initialized: now 1786451740959959 us; error 41 us; skew 500 ppm
I20260811 12:35:40.960376 24051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:35:40 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:35:40.963624 24051 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:35:40.965794 24051 webserver.cc:460] Webserver started at http://127.3.204.66:42431/ using document root <none> and password file <none>
I20260811 12:35:40.966570 24051 fs_manager.cc:359] Metadata directory not provided
I20260811 12:35:40.966701 24051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:35:40.967141 24051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 12:35:40 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:35:40.987736 24051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "2041a7459d1f456389ad9660ece18cb3"
format_stamp: "Formatted at 2026-08-11 12:35:40 on dist-test-slave-lgr5"
server_key: "805e484183fe276e18584ebcde3c7843"
server_key_iv: "458ac3c8c7bd98459888936051590d48"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:40.989004 24051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "2041a7459d1f456389ad9660ece18cb3"
format_stamp: "Formatted at 2026-08-11 12:35:40 on dist-test-slave-lgr5"
server_key: "805e484183fe276e18584ebcde3c7843"
server_key_iv: "458ac3c8c7bd98459888936051590d48"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:40.996953 24051 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260811 12:35:41.018203 24087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:41.020102 24051 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260811 12:35:41.020368 24051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "2041a7459d1f456389ad9660ece18cb3"
format_stamp: "Formatted at 2026-08-11 12:35:40 on dist-test-slave-lgr5"
server_key: "805e484183fe276e18584ebcde3c7843"
server_key_iv: "458ac3c8c7bd98459888936051590d48"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:41.020684 24051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:35:41.044852 24051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:35:41.051800 24051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:35:41.052443 24051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:35:41.054221 24051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:35:41.056993 24051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:35:41.057153 24051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:41.057369 24051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:35:41.057518 24051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:41.120018 24040 heartbeater.cc:503] Master 127.3.204.126:37709 was elected leader, sending a full tablet report...
I20260811 12:35:41.142778 24051 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:46247
I20260811 12:35:41.142886 24282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:46247 every 8 connection(s)
I20260811 12:35:41.144708 24051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:35:41.151598 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 24051
I20260811 12:35:41.151813 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:35:41.152597 3889 external_mini_cluster.cc:1428] Setting key aa74626ba9d40d4432726496f4165269
Aug 11 12:35:41 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:35:41.177497 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:41.148674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:48711 (local address 127.3.204.126:37709)
0811 12:35:41.149477 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:41.149481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:41.150747 (+ 1266us) server_negotiation.cc:407] Connection header received
0811 12:35:41.154363 (+ 3616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:41.154367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:41.154440 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:41.154564 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:35:41.157331 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:41.158594 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:41.161676 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:41.162257 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:41.162360 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:41.167875 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:35:41.167881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:35:41.167885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:35:41.167951 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:35:41.173018 (+ 5067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:41.174069 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:41.174075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:41.174079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:41.174342 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:41.174964 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:41.174969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:41.174972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:41.175475 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:35:41.175625 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:41.176683 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0811 12:35:41.176917 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":482,"threads_started":1}
I20260811 12:35:41.178422 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:41.149391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37709 (local address 127.3.204.66:48711)
0811 12:35:41.150564 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0811 12:35:41.150612 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 12:35:41.153969 (+ 3357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:35:41.154879 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:35:41.154891 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:35:41.155802 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:35:41.157115 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:41.157148 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:41.158770 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:41.158775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:41.161456 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:41.161504 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:41.162571 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:41.162580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:41.162913 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:41.163891 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:35:41.163929 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:35:41.167475 (+ 3546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:35:41.173235 (+ 5760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:35:41.173245 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:35:41.173262 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:35:41.173903 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:35:41.174490 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:35:41.174494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:35:41.174497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:35:41.174829 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:35:41.175647 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:35:41.175655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:35:41.176147 (+ 492us) client_negotiation.cc:769] Sending connection context
0811 12:35:41.176507 (+ 360us) client_negotiation.cc:240] Negotiation successful
0811 12:35:41.177053 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":606,"threads_started":1}
I20260811 12:35:41.180513 24285 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37709
I20260811 12:35:41.180887 24285 heartbeater.cc:464] Registering TS with master...
I20260811 12:35:41.181979 24285 heartbeater.cc:511] Master 127.3.204.126:37709 requested a full tablet report, sending...
I20260811 12:35:41.183902 23622 ts_manager.cc:194] Registered new tserver with Master: 2041a7459d1f456389ad9660ece18cb3 (127.3.204.66:46247)
I20260811 12:35:41.184952 23622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:48711
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:35:41.223876 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:59051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37709
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:35:41.873821 24296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:35:41.874210 24296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:35:41.874433 24296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:35:41.885169 24296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:35:41.885409 24296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:35:41.885645 24296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:35:41.899125 24296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.204.84:59051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:35:41.900722 24296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:35:41.902241 24296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:35:41.919900 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:35:41.920426 24308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:35:41.920591 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:35:41.922297 24296 server_base.cc:1034] running on GCE node
I20260811 12:35:41.922956 24296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:35:41.924284 24296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:35:41.925595 24296 hybrid_clock.cc:648] HybridClock initialized: now 1786451741925478 us; error 55 us; skew 500 ppm
I20260811 12:35:41.926075 24296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:35:41 dist-test-slave-lgr5 krb5kdc[22700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:35:41.930781 24296 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:35:41.932816 24296 webserver.cc:460] Webserver started at http://127.3.204.67:41621/ using document root <none> and password file <none>
I20260811 12:35:41.933668 24296 fs_manager.cc:359] Metadata directory not provided
I20260811 12:35:41.933863 24296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:35:41.934333 24296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 12:35:41 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:35:41.951767 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "32ca83a059684480beada88183b67734"
format_stamp: "Formatted at 2026-08-11 12:35:41 on dist-test-slave-lgr5"
server_key: "71605ff26e00df9fc6b29e3090a73169"
server_key_iv: "9692c49c90f8ce220814f8cd805a09af"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:41.952767 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "32ca83a059684480beada88183b67734"
format_stamp: "Formatted at 2026-08-11 12:35:41 on dist-test-slave-lgr5"
server_key: "71605ff26e00df9fc6b29e3090a73169"
server_key_iv: "9692c49c90f8ce220814f8cd805a09af"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:41.960557 24296 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260811 12:35:41.981750 24319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:41.983572 24296 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 12:35:41.983835 24296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "32ca83a059684480beada88183b67734"
format_stamp: "Formatted at 2026-08-11 12:35:41 on dist-test-slave-lgr5"
server_key: "71605ff26e00df9fc6b29e3090a73169"
server_key_iv: "9692c49c90f8ce220814f8cd805a09af"
server_key_version: "kuduclusterkey@0"
I20260811 12:35:41.984146 24296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:35:42.004230 24296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:35:42.010171 24296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:35:42.010694 24296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:35:42.012293 24296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:35:42.014659 24296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:35:42.014817 24296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:42.015043 24296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:35:42.015158 24296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:35:42.097056 24296 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:38579
I20260811 12:35:42.097157 24514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:38579 every 8 connection(s)
I20260811 12:35:42.098838 24296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:35:42.102458 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 24296
I20260811 12:35:42.102675 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:35:42.103331 3889 external_mini_cluster.cc:1428] Setting key 5b4a75d8442af5b5ec98b41aba8d1b43
Aug 11 12:35:42 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:35:42.128001 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.102291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:40605 (local address 127.3.204.126:37709)
0811 12:35:42.103342 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.103348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.104746 (+ 1398us) server_negotiation.cc:407] Connection header received
0811 12:35:42.108098 (+ 3352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.108105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.108189 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.108282 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:35:42.110325 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.111812 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.114120 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.114838 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.114958 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.119960 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:35:42.119967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:35:42.119972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:35:42.120049 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:35:42.124302 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.125265 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.125270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.125272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.125447 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.125988 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.125992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.125994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.126465 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:35:42.126634 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.127360 (+ 726us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.127552 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":697,"threads_started":1}
I20260811 12:35:42.129228 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.103001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37709 (local address 127.3.204.67:40605)
0811 12:35:42.104548 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0811 12:35:42.104617 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0811 12:35:42.107795 (+ 3178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:35:42.108539 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:35:42.108547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:35:42.109196 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:35:42.110139 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.110161 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.112020 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.112025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.113931 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.113957 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.115155 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.115162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.115465 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.116327 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:35:42.116354 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:35:42.119657 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:35:42.124481 (+ 4824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:35:42.124493 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:35:42.124513 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:35:42.125120 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:35:42.125559 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:35:42.125562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:35:42.125565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:35:42.125873 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:35:42.126606 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:35:42.126613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:35:42.126995 (+ 382us) client_negotiation.cc:769] Sending connection context
0811 12:35:42.127270 (+ 275us) client_negotiation.cc:240] Negotiation successful
0811 12:35:42.127802 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"thread_start_us":794,"threads_started":1}
I20260811 12:35:42.130790 24515 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37709
I20260811 12:35:42.131199 24515 heartbeater.cc:464] Registering TS with master...
I20260811 12:35:42.132464 24515 heartbeater.cc:511] Master 127.3.204.126:37709 requested a full tablet report, sending...
I20260811 12:35:42.134262 23622 ts_manager.cc:194] Registered new tserver with Master: 32ca83a059684480beada88183b67734 (127.3.204.67:38579)
I20260811 12:35:42.135385 23622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:40605
I20260811 12:35:42.145767 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:35:42.170887 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.152534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.3.204.126:37709)
0811 12:35:42.152758 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.152765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.153649 (+ 884us) server_negotiation.cc:407] Connection header received
0811 12:35:42.153822 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.153826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.153947 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.154041 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:35:42.156331 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.157412 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.159322 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.160135 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.160260 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.163932 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:35:42.163937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:35:42.163940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:35:42.163979 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:35:42.167510 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.168499 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.168504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.168507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.168700 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.169061 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.169064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.169066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.169526 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:35:42.169856 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.170236 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.170466 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:35:42.175863 23622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:35:42.187700 23622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:35:42.188189 24285 heartbeater.cc:503] Master 127.3.204.126:37709 was elected leader, sending a full tablet report...
I20260811 12:35:42.234678 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.205833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.3.204.66:46247)
0811 12:35:42.218945 (+ 13112us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.218953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.221435 (+ 2482us) server_negotiation.cc:407] Connection header received
0811 12:35:42.221791 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.221799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.222071 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.222267 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:35:42.224496 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.226837 (+ 2341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.230996 (+ 4159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.232730 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.232933 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.233256 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.233701 (+ 445us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.233950 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12904,"thread_start_us":12731,"threads_started":1}
I20260811 12:35:42.234704 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.205894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.3.204.67:38579)
0811 12:35:42.217754 (+ 11860us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.217767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.220403 (+ 2636us) server_negotiation.cc:407] Connection header received
0811 12:35:42.221220 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.221231 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.221596 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.221833 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:35:42.223840 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.226152 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.230924 (+ 4772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.233067 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.233213 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.233580 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.233927 (+ 347us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.234155 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11558,"thread_start_us":11931,"threads_started":1}
I20260811 12:35:42.236356 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.207189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.3.204.65:38467)
0811 12:35:42.220337 (+ 13148us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.220346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.222507 (+ 2161us) server_negotiation.cc:407] Connection header received
0811 12:35:42.222671 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.222680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.222893 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.223157 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:35:42.224718 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.227027 (+ 2309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.231259 (+ 4232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.233428 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.233597 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.233857 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.235708 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.235868 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12928,"thread_start_us":12696,"threads_started":1}
I20260811 12:35:42.236665 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.206569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38467 (local address 127.0.0.1:54364)
0811 12:35:42.222189 (+ 15620us) negotiation.cc:107] Waiting for socket to connect
0811 12:35:42.222208 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:35:42.222372 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:35:42.223138 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:35:42.223142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:35:42.223182 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:35:42.224497 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.224511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.227237 (+ 2726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.227242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.231010 (+ 3768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.231026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.234176 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.234185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.234567 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.234601 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 12:35:42.234688 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 12:35:42.234941 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15396,"mutex_wait_us":130,"spinlock_wait_cycles":3840,"thread_start_us":14277,"threads_started":1}
I20260811 12:35:42.236775 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.205384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:38579 (local address 127.0.0.1:59336)
0811 12:35:42.220179 (+ 14795us) negotiation.cc:107] Waiting for socket to connect
0811 12:35:42.220230 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 12:35:42.220669 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:35:42.222414 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:35:42.222423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:35:42.222492 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:35:42.223405 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.223422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.226358 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.226364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.230719 (+ 4355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.230742 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.233492 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.233499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.233700 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.233732 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 12:35:42.235039 (+ 1307us) client_negotiation.cc:240] Negotiation successful
0811 12:35:42.235182 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14417,"mutex_wait_us":42,"thread_start_us":16334,"threads_started":1}
I20260811 12:35:42.237300 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.205600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46247 (local address 127.0.0.1:58016)
0811 12:35:42.221304 (+ 15704us) negotiation.cc:107] Waiting for socket to connect
0811 12:35:42.221327 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:35:42.221575 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:35:42.222712 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:35:42.222717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:35:42.222743 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:35:42.224142 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.224165 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.227006 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.227011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.230815 (+ 3804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.230829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.233084 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.233092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.233372 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.233454 (+ 82us) client_negotiation.cc:769] Sending connection context
0811 12:35:42.235270 (+ 1816us) client_negotiation.cc:240] Negotiation successful
0811 12:35:42.235477 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15524,"mutex_wait_us":71,"spinlock_wait_cycles":12416,"thread_start_us":13781,"threads_started":1}
I20260811 12:35:42.241626 23920 tablet_service.cc:1467] Processing CreateTablet for tablet 6b792b9ecf9b495a9c48916537fdfb7c (DEFAULT_TABLE table=test-table [id=5d268e68842f4c5fa9468e44c1c89f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:35:42.241627 24395 tablet_service.cc:1467] Processing CreateTablet for tablet 6b792b9ecf9b495a9c48916537fdfb7c (DEFAULT_TABLE table=test-table [id=5d268e68842f4c5fa9468e44c1c89f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:35:42.243305 24395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b792b9ecf9b495a9c48916537fdfb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:35:42.243470 23920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b792b9ecf9b495a9c48916537fdfb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:35:42.245889 24169 tablet_service.cc:1467] Processing CreateTablet for tablet 6b792b9ecf9b495a9c48916537fdfb7c (DEFAULT_TABLE table=test-table [id=5d268e68842f4c5fa9468e44c1c89f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:35:42.247668 24169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b792b9ecf9b495a9c48916537fdfb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:35:42.260644 24559 tablet_bootstrap.cc:492] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Bootstrap starting.
I20260811 12:35:42.264689 24561 tablet_bootstrap.cc:492] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Bootstrap starting.
I20260811 12:35:42.267709 24559 tablet_bootstrap.cc:654] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Neither blocks nor log segments found. Creating new log.
I20260811 12:35:42.268565 24563 tablet_bootstrap.cc:492] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Bootstrap starting.
I20260811 12:35:42.270360 24559 log.cc:824] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Log is configured to *not* fsync() on all Append() calls
I20260811 12:35:42.271185 24561 tablet_bootstrap.cc:654] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Neither blocks nor log segments found. Creating new log.
I20260811 12:35:42.273257 24561 log.cc:824] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Log is configured to *not* fsync() on all Append() calls
I20260811 12:35:42.274350 24563 tablet_bootstrap.cc:654] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Neither blocks nor log segments found. Creating new log.
I20260811 12:35:42.275748 24559 tablet_bootstrap.cc:492] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: No bootstrap required, opened a new log
I20260811 12:35:42.276207 24559 ts_tablet_manager.cc:1397] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20260811 12:35:42.276682 24563 log.cc:824] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Log is configured to *not* fsync() on all Append() calls
I20260811 12:35:42.278055 24561 tablet_bootstrap.cc:492] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: No bootstrap required, opened a new log
I20260811 12:35:42.278522 24561 ts_tablet_manager.cc:1397] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260811 12:35:42.281178 24563 tablet_bootstrap.cc:492] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: No bootstrap required, opened a new log
I20260811 12:35:42.281644 24563 ts_tablet_manager.cc:1397] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.004s
I20260811 12:35:42.293483 24563 raft_consensus.cc:348] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.293514 24559 raft_consensus.cc:348] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.294046 24563 raft_consensus.cc:374] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:35:42.294126 24559 raft_consensus.cc:374] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:35:42.294214 24563 raft_consensus.cc:729] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2041a7459d1f456389ad9660ece18cb3, State: Initialized, Role: FOLLOWER
I20260811 12:35:42.294322 24559 raft_consensus.cc:729] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ca83a059684480beada88183b67734, State: Initialized, Role: FOLLOWER
I20260811 12:35:42.294215 24561 raft_consensus.cc:348] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.295028 24561 raft_consensus.cc:374] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:35:42.295275 24561 raft_consensus.cc:729] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881f5b25bcae4952a9c1358aecc3796a, State: Initialized, Role: FOLLOWER
I20260811 12:35:42.295112 24563 consensus_queue.cc:260] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.295105 24559 consensus_queue.cc:260] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.296303 24561 consensus_queue.cc:260] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.298507 24563 ts_tablet_manager.cc:1428] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260811 12:35:42.300900 24515 heartbeater.cc:503] Master 127.3.204.126:37709 was elected leader, sending a full tablet report...
I20260811 12:35:42.302098 24561 ts_tablet_manager.cc:1428] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Time spent starting tablet: real 0.023s user 0.012s sys 0.012s
I20260811 12:35:42.302371 24559 ts_tablet_manager.cc:1428] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Time spent starting tablet: real 0.026s user 0.003s sys 0.020s
W20260811 12:35:42.326783 24042 tablet.cc:2369] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:35:42.351761 24521 tablet.cc:2369] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:35:42.353714 24570 raft_consensus.cc:482] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:35:42.354118 24570 raft_consensus.cc:504] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.356444 24570 leader_election.cc:288] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2041a7459d1f456389ad9660ece18cb3 (127.3.204.66:46247), 881f5b25bcae4952a9c1358aecc3796a (127.3.204.65:38467)
I20260811 12:35:42.364409 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.357387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:58343 (local address 127.3.204.66:46247)
0811 12:35:42.357648 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.357653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.357729 (+ 76us) server_negotiation.cc:407] Connection header received
0811 12:35:42.358055 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.358062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.358152 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.358269 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:35:42.358743 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.359733 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.362107 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.363291 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.363483 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.363567 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.363938 (+ 371us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.364062 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 12:35:42.364446 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.357282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:46247 (local address 127.3.204.67:58343)
0811 12:35:42.357636 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0811 12:35:42.357658 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:35:42.357868 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:35:42.358280 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:35:42.358283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:35:42.358309 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:35:42.358594 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.358603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.359877 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.359881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.361923 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.361936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.363437 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.363444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.363744 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.363770 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 12:35:42.363885 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 12:35:42.364034 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":23}
I20260811 12:35:42.366187 24209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b792b9ecf9b495a9c48916537fdfb7c" candidate_uuid: "32ca83a059684480beada88183b67734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2041a7459d1f456389ad9660ece18cb3" is_pre_election: true
I20260811 12:35:42.367242 24209 raft_consensus.cc:2454] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ca83a059684480beada88183b67734 in term 0.
I20260811 12:35:42.367906 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.357376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:38761 (local address 127.3.204.65:38467)
0811 12:35:42.357639 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.357647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.359620 (+ 1973us) server_negotiation.cc:407] Connection header received
0811 12:35:42.359861 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.359866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.359932 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.360018 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:35:42.361225 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.362184 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.365246 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.366832 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.367082 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.367165 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.367446 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.367593 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 12:35:42.368281 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.358182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38467 (local address 127.3.204.67:38761)
0811 12:35:42.359501 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0811 12:35:42.359518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:35:42.359677 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:35:42.360053 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:35:42.360059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:35:42.360109 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:35:42.361009 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.361027 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.362398 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.362402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.365042 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:35:42.365061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.367138 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.367144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:35:42.367284 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.367310 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 12:35:42.367390 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 12:35:42.367486 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":5,"thread_start_us":884,"threads_started":1}
I20260811 12:35:42.368608 24324 leader_election.cc:302] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2041a7459d1f456389ad9660ece18cb3, 32ca83a059684480beada88183b67734; no voters:
I20260811 12:35:42.369287 24570 raft_consensus.cc:2781] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:35:42.369589 24570 raft_consensus.cc:482] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:35:42.369817 24570 raft_consensus.cc:3037] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:35:42.370050 23960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b792b9ecf9b495a9c48916537fdfb7c" candidate_uuid: "32ca83a059684480beada88183b67734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881f5b25bcae4952a9c1358aecc3796a" is_pre_election: true
I20260811 12:35:42.371065 23960 raft_consensus.cc:2454] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ca83a059684480beada88183b67734 in term 0.
I20260811 12:35:42.374197 24570 raft_consensus.cc:504] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.375144 24570 leader_election.cc:288] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [CANDIDATE]: Term 1 election: Requested vote from peers 2041a7459d1f456389ad9660ece18cb3 (127.3.204.66:46247), 881f5b25bcae4952a9c1358aecc3796a (127.3.204.65:38467)
I20260811 12:35:42.375749 23960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b792b9ecf9b495a9c48916537fdfb7c" candidate_uuid: "32ca83a059684480beada88183b67734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881f5b25bcae4952a9c1358aecc3796a"
I20260811 12:35:42.375770 24209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b792b9ecf9b495a9c48916537fdfb7c" candidate_uuid: "32ca83a059684480beada88183b67734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2041a7459d1f456389ad9660ece18cb3"
I20260811 12:35:42.376039 24209 raft_consensus.cc:3037] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:35:42.376060 23960 raft_consensus.cc:3037] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:35:42.380930 23960 raft_consensus.cc:2454] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ca83a059684480beada88183b67734 in term 1.
I20260811 12:35:42.380930 24209 raft_consensus.cc:2454] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ca83a059684480beada88183b67734 in term 1.
I20260811 12:35:42.381618 24321 leader_election.cc:302] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ca83a059684480beada88183b67734, 881f5b25bcae4952a9c1358aecc3796a; no voters:
I20260811 12:35:42.382098 24570 raft_consensus.cc:2781] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:35:42.382968 24570 raft_consensus.cc:686] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 1 LEADER]: Becoming Leader. State: Replica: 32ca83a059684480beada88183b67734, State: Running, Role: LEADER
I20260811 12:35:42.383510 24570 consensus_queue.cc:237] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } }
I20260811 12:35:42.390029 23620 catalog_manager.cc:5495] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32ca83a059684480beada88183b67734 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "32ca83a059684480beada88183b67734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ca83a059684480beada88183b67734" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 } health_report { overall_health: UNKNOWN } } }
W20260811 12:35:42.398205 24286 tablet.cc:2369] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:35:42.441097 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.434066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.3.204.67:38579)
0811 12:35:42.434310 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.434316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.434338 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:35:42.434467 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.434471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.434599 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.434741 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:35:42.435362 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.436502 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.438461 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.439007 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.439125 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.439838 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:35:42.440436 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:35:42.440577 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.440680 (+ 103us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.440767 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:35:42.452473 23960 raft_consensus.cc:1261] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 1 FOLLOWER]: Refusing update from remote peer 32ca83a059684480beada88183b67734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:35:42.452533 24209 raft_consensus.cc:1261] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 1 FOLLOWER]: Refusing update from remote peer 32ca83a059684480beada88183b67734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:35:42.453706 24574 consensus_queue.cc:1035] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2041a7459d1f456389ad9660ece18cb3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:35:42.454342 24570 consensus_queue.cc:1035] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881f5b25bcae4952a9c1358aecc3796a" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:35:42.488509 23620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:35:42.489073 23620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:35:42.494212 23620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd75c58bf8dd4502b0bbe94fbf37e019: Sending DeleteTablet for 3 replicas of tablet 6b792b9ecf9b495a9c48916537fdfb7c
I20260811 12:35:42.495985 24395 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b792b9ecf9b495a9c48916537fdfb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:35:42 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:59336
I20260811 12:35:42.496021 23920 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b792b9ecf9b495a9c48916537fdfb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:35:42 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:54364
I20260811 12:35:42.496194 24169 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b792b9ecf9b495a9c48916537fdfb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:35:42 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:58016
I20260811 12:35:42.498719 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 23799
I20260811 12:35:42.508726 24600 tablet_replica.cc:330] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: stopping tablet replica
I20260811 12:35:42.509280 24600 raft_consensus.cc:2229] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:35:42.510042 24600 raft_consensus.cc:2258] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:35:42.511008 24599 tablet_replica.cc:330] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: stopping tablet replica
I20260811 12:35:42.511668 24599 raft_consensus.cc:2229] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:35:42.513011 24599 raft_consensus.cc:2258] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:35:42.515950 24599 ts_tablet_manager.cc:1905] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:35:42.516022 24600 ts_tablet_manager.cc:1905] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:35:42.517755 24598 tablet_replica.cc:330] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: stopping tablet replica
I20260811 12:35:42.518292 24598 raft_consensus.cc:2229] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:35:42.518985 24598 raft_consensus.cc:2258] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:35:42.524999 24600 ts_tablet_manager.cc:1918] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:35:42.525290 24600 log.cc:1196] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/6b792b9ecf9b495a9c48916537fdfb7c
I20260811 12:35:42.525998 24600 ts_tablet_manager.cc:1939] T 6b792b9ecf9b495a9c48916537fdfb7c P 2041a7459d1f456389ad9660ece18cb3: Deleting consensus metadata
Aug 11 12:35:42 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:35:42.528275 23601 catalog_manager.cc:4841] TS 2041a7459d1f456389ad9660ece18cb3 (127.3.204.66:46247): tablet 6b792b9ecf9b495a9c48916537fdfb7c (table test-table [id=5d268e68842f4c5fa9468e44c1c89f93]) successfully deleted
I20260811 12:35:42.528755 24598 ts_tablet_manager.cc:1905] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:35:42.530094 24599 ts_tablet_manager.cc:1918] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:35:42.530319 24599 log.cc:1196] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/6b792b9ecf9b495a9c48916537fdfb7c
I20260811 12:35:42.531041 24599 ts_tablet_manager.cc:1939] T 6b792b9ecf9b495a9c48916537fdfb7c P 32ca83a059684480beada88183b67734: Deleting consensus metadata
I20260811 12:35:42.533191 23598 catalog_manager.cc:4841] TS 32ca83a059684480beada88183b67734 (127.3.204.67:38579): tablet 6b792b9ecf9b495a9c48916537fdfb7c (table test-table [id=5d268e68842f4c5fa9468e44c1c89f93]) successfully deleted
I20260811 12:35:42.538704 24598 ts_tablet_manager.cc:1918] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:35:42.538965 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.500314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.3.204.65:38467)
0811 12:35:42.500514 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.500520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.509084 (+ 8564us) server_negotiation.cc:407] Connection header received
0811 12:35:42.509421 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.509427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.509524 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.509609 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:35:42.513130 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.516894 (+ 3764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.519501 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.520817 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.520994 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.528410 (+ 7416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:35:42.528429 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:35:42.528447 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:35:42.528502 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:35:42.534604 (+ 6102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.535768 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.535780 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.535783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.536053 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.536700 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.536705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.536709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.537272 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:35:42.537609 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.538001 (+ 392us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.538297 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:35:42.538986 24598 log.cc:1196] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/6b792b9ecf9b495a9c48916537fdfb7c
I20260811 12:35:42.539857 24598 ts_tablet_manager.cc:1939] T 6b792b9ecf9b495a9c48916537fdfb7c P 881f5b25bcae4952a9c1358aecc3796a: Deleting consensus metadata
I20260811 12:35:42.539932 24034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:35:42.632571 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 23799
I20260811 12:35:42.632661 23599 catalog_manager.cc:4841] TS 881f5b25bcae4952a9c1358aecc3796a (127.3.204.65:38467): tablet 6b792b9ecf9b495a9c48916537fdfb7c (table test-table [id=5d268e68842f4c5fa9468e44c1c89f93]) successfully deleted
I20260811 12:35:42.662773 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 24051
Aug 11 12:35:42 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:35:42.681470 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.663685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.3.204.66:46247)
0811 12:35:42.663905 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.663912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.663936 (+ 24us) server_negotiation.cc:407] Connection header received
0811 12:35:42.664028 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.664034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.664166 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.664271 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:35:42.665568 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.666713 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.668235 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.668906 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.669048 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.674102 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:35:42.674125 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:35:42.674149 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:35:42.674202 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:35:42.678203 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.678991 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.679002 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.679006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.679182 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.679615 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.679620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.679623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.680190 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:35:42.680493 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.680789 (+ 296us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.681063 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:35:42.682276 24271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:35:42.772341 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 24051
I20260811 12:35:42.795917 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 24296
Aug 11 12:35:42 dist-test-slave-lgr5 krb5kdc[22700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:35:42.816862 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:35:42.796849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.3.204.67:38579)
0811 12:35:42.797011 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 12:35:42.797016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:35:42.797038 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:35:42.797138 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:35:42.797142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:35:42.797209 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:35:42.797301 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:35:42.799243 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.800588 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.802504 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:35:42.803232 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:35:42.803357 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:35:42.809192 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:35:42.809209 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:35:42.809232 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:35:42.809282 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:35:42.813304 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.814254 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.814263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.814266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.814485 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:35:42.814981 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:35:42.814999 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:35:42.815003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:35:42.815541 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:35:42.815887 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0811 12:35:42.816209 (+ 322us) server_negotiation.cc:299] Negotiation successful
0811 12:35:42.816474 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 12:35:42.817752 24510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:35:42.940184 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 24296
I20260811 12:35:42.973289 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 23574
I20260811 12:35:42.974637 23690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:35:43.089681 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 23574
I20260811 12:35:43.127300 3889 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 12:35:43.473215 3889 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 12:35:43.473527 3889 mini_ranger.cc:66] Stopping Ranger...
I20260811 12:35:43.819412 3889 mini_ranger.cc:68] Stopped Ranger
2026-08-11 12:35:43.819 UTC [22754] LOG: received smart shutdown request
2026-08-11 12:35:43.824 UTC [22754] LOG: background worker "logical replication launcher" (PID 22772) exited with exit code 1
2026-08-11 12:35:43.824 UTC [22763] LOG: shutting down
2026-08-11 12:35:43.999 UTC [22754] LOG: database system is shut down
2026-08-11T12:35:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52737 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:35:44 dist-test-slave-lgr5 krb5kdc[24624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:35:44 dist-test-slave-lgr5 krb5kdc[24624](info): set up 2 sockets
Aug 11 12:35:44 dist-test-slave-lgr5 krb5kdc[24624](info): commencing operation
krb5kdc: starting...
W20260811 12:35:45.393465 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:35:45 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:35:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/rangeradmin_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/rangeradmin_127.3.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/rangerlookup_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/rangerlookup_127.3.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/HTTP_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/HTTP_127.3.204.84@KRBTEST.COM.keytab.
I20260811 12:35:45.786913 3889 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1nKHSZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/postgres -l logfile start
2026-08-11 12:35:48.380 UTC [24677] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 12:35:48.380 UTC [24677] LOG: listening on IPv4 address "127.3.204.84", port 60835
2026-08-11 12:35:48.385 UTC [24677] LOG: listening on Unix socket "/tmp/.s.PGSQL.60835"
2026-08-11 12:35:48.406 UTC [24685] LOG: database system was shut down at 2026-08-11 12:35:46 UTC
2026-08-11 12:35:48.410 UTC [24677] LOG: database system is ready to accept connections
I20260811 12:35:49.532732 3889 mini_postgres.cc:96] Postgres bound to 60835
2026-08-11 12:35:49.555 UTC [24700] FATAL: database "slave" does not exist
127.3.204.84:60835 - accepting connections
I20260811 12:35:49.555913 3889 mini_ranger.cc:161] Starting Ranger...
I20260811 12:35:49.588337 3889 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 12:35:50.117908 3889 mini_ranger.cc:87] Created ranger Postgres database
I20260811 12:35:50.118022 3889 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-admin
2026-08-11 12:35:50,412 [I] DB FLAVOR :POSTGRES
2026-08-11 12:35:50,413 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 12:35:50,413 [I] Checking connection..
2026-08-11 12:35:50,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 12:35:50,687 [I] Checking connection passed.
2026-08-11 12:35:50,688 [I] --------- Verifying version history table ---------
2026-08-11 12:35:50,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 12:35:50,952 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 12:35:50,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 12:35:51,202 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 12:35:51,203 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 12:35:51,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 12:35:51.444 UTC [24812] WARNING: there is no transaction in progress
2026-08-11 12:35:51,451 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 12:35:51,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 12:35:51,719 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 12:35:51,720 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 12:35:51,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 12:35:51,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 12:35:52,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lgr5.c.gcp-upstream.internal','N') ;"
2026-08-11 12:35:52,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 12:35:52,755 [I] Table x_portal_user does not exist in database ranger
2026-08-11 12:35:52,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 12:35:53,024 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 12:35:53,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 12:35:53,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 12:35:53,545 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 12:35:53,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 12:35:54.581 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.599 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.616 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.632 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.648 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.665 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.721 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.729 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.744 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.755 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.767 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.781 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.791 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.797 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.805 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:54.812 UTC [24989] WARNING: there is no transaction in progress
2026-08-11 12:35:55,516 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 12:35:55,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 12:35:55,774 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 12:35:55,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 12:35:56,025 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 12:35:56,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 12:35:56,301 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 12:35:56,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 12:35:56,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 12:35:56,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lgr5.c.gcp-upstream.internal';"
2026-08-11 12:35:57,073 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 12:35:57,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 12:35:57,319 [I] DB_PATCHES have already been applied
I20260811 12:35:57.325646 3889 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-admin:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1nKHSZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 12:35:57.325738 3889 mini_ranger.cc:193] Using host: 127.3.204.84
I20260811 12:35:57.345649 3889 mini_ranger.cc:227] Ranger admin URL: http://127.3.204.84:49895
Aug 11, 2026 12:35:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 12:35:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 12:35:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 12:35:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 12:35:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 12:35:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 12:35:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 12:35:58.636670 3889 mini_ranger.cc:160] Time spent starting Ranger: real 9.081s user 0.002s sys 0.060s
Aug 11, 2026 12:36:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 12:36:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 12:36:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 12:36:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 12:36:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 12:36:19.611023 3889 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/rangerkms_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/rangerkms_127.3.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.204.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.204.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.204.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/HTTP_127.3.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.204.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/HTTP_127.3.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 12:36:19 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451779, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 12:36:19.818321 3889 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 12:36:19.845142 3889 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 12:36:20.765115 3889 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 12:36:20.765235 3889 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms
I20260811 12:36:20.765277 3889 mini_ranger_kms.cc:227] Using postgres at 127.3.204.84:60835
2026-08-11 12:36:21,149 [I] DB FLAVOR :POSTGRES
2026-08-11 12:36:21,149 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 12:36:21,149 [I] Checking connection
2026-08-11 12:36:21,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 12:36:21,391 [I] connection success
2026-08-11 12:36:21,391 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 12:36:21,391 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 12:36:21,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 12:36:21,639 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 12:36:21,639 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 12:36:21,639 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 12:36:21,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.204.84:60835/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 12:36:21,921 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 12:36:22.407343 3889 mini_ranger_kms.cc:310] Created kms service
I20260811 12:36:22.582567 3889 mini_ranger_kms.cc:326] Created kudu user
I20260811 12:36:22.626951 3889 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 12:36:22 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451779, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:23.054708 3889 mini_ranger_kms.cc:384] Added ranger policy
I20260811 12:36:23.055032 3889 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms:/tmp/dist-test-task1nKHSZ/build/asan/bin/postgresql.jar:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1nKHSZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 12:36:23.055075 3889 mini_ranger_kms.cc:240] Using host: 127.3.204.84
I20260811 12:36:23.079818 3889 mini_ranger_kms.cc:276] Ranger KMS PID: 25333
I20260811 12:36:23.079947 3889 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.204.84:46261
Aug 11, 2026 12:36:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 12:36:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 12:36:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 12:36:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 12:36:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 12:36:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 12:36:25 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786451785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.204.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 11 12:36:25 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.204.84@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:25.806099 3889 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.988s user 0.051s sys 0.370s
I20260811 12:36:25.806345 3889 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 12:36:25.806526 3889 mini_ranger_kms.cc:401] 127.3.204.84:46261/kms/v1/keys
Aug 11 12:36:26 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.204.84@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
Aug 11 12:36:28 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:36:28.468214 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:45567
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:45567
--ranger_config_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:46261/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:28.811383 25418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:28.811681 25418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:28.811771 25418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:28.819589 25418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:28.819726 25418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:28.819770 25418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:28.819824 25418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:28.819870 25418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:28.830430 25418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.204.84:46261/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:45567
--ranger_config_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:28.831753 25418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:28.833112 25418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:28.845357 25429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:28.845463 25430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:28.845463 25432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:28.846388 25418 server_base.cc:1034] running on GCE node
I20260811 12:36:28.847205 25418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:28.848729 25418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:28.849924 25418 hybrid_clock.cc:648] HybridClock initialized: now 1786451788849884 us; error 32 us; skew 500 ppm
I20260811 12:36:28.850278 25418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:28 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451788, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:28.859967 25418 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:36:28.861644 25418 webserver.cc:460] Webserver started at http://127.3.204.126:38233/ using document root <none> and password file <none>
I20260811 12:36:28.862335 25418 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:28.862438 25418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:28.862908 25418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 12:36:28 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451788, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:28.931553 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "0731f978074042b0a15a0370c96fdb67"
format_stamp: "Formatted at 2026-08-11 12:36:28 on dist-test-slave-lgr5"
server_key: "5e4afea9b06ec35620a0b726a8251ee4"
server_key_iv: "a5cfe054d67e51cd8c2a4e09171ae2e3"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:28.932538 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "0731f978074042b0a15a0370c96fdb67"
format_stamp: "Formatted at 2026-08-11 12:36:28 on dist-test-slave-lgr5"
server_key: "5e4afea9b06ec35620a0b726a8251ee4"
server_key_iv: "a5cfe054d67e51cd8c2a4e09171ae2e3"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:28.938301 25418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:36:28.959125 25444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:28.960686 25418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:36:28.960880 25418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "0731f978074042b0a15a0370c96fdb67"
format_stamp: "Formatted at 2026-08-11 12:36:28 on dist-test-slave-lgr5"
server_key: "5e4afea9b06ec35620a0b726a8251ee4"
server_key_iv: "a5cfe054d67e51cd8c2a4e09171ae2e3"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:28.961108 25418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:28.997762 25418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:29.002058 25418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:29.002401 25418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:29.021682 25418 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:45567
I20260811 12:36:29.021685 25541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:45567 every 8 connection(s)
I20260811 12:36:29.023111 25418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:36:29.027057 25545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:29.030444 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 25418
I20260811 12:36:29.030663 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:36:29.031486 3889 external_mini_cluster.cc:1428] Setting key 7460d4839a44e97c0a8a9d0c820f34ce
I20260811 12:36:29.041458 25545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Bootstrap starting.
I20260811 12:36:29.046231 25545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:29.047844 25545 log.cc:824] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:29.051236 25545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: No bootstrap required, opened a new log
I20260811 12:36:29.060693 25545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } }
I20260811 12:36:29.061264 25545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:29.061386 25545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0731f978074042b0a15a0370c96fdb67, State: Initialized, Role: FOLLOWER
I20260811 12:36:29.062186 25545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } }
I20260811 12:36:29.062489 25545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:36:29.062685 25545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:36:29.062881 25545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 0 FOLLOWER]: Advancing to term 1
Aug 11 12:36:29 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:29.065670 25545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } }
I20260811 12:36:29.066411 25545 leader_election.cc:302] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0731f978074042b0a15a0370c96fdb67; no voters:
I20260811 12:36:29.067184 25545 leader_election.cc:288] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:36:29.067320 25554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:36:29.076970 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:29.033663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.3.204.126:45567)
0811 12:36:29.048435 (+ 14772us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:29.048453 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:29.048509 (+ 56us) server_negotiation.cc:407] Connection header received
0811 12:36:29.051267 (+ 2758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:29.051301 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:29.051855 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:29.052484 (+ 629us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:29.054184 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:29.056452 (+ 2268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:29.058454 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:29.059602 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:29.059795 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:29.064990 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:29.065002 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:29.065029 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:29.065082 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:29.070050 (+ 4968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:29.071147 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:29.071158 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:29.071169 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:29.071378 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:29.071875 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:29.071881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:29.071884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:29.072787 (+ 903us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:29.074360 (+ 1573us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:29.074720 (+ 360us) server_negotiation.cc:299] Negotiation successful
0811 12:36:29.075258 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14488,"thread_start_us":14942,"threads_started":1}
I20260811 12:36:29.081015 25554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 1 LEADER]: Becoming Leader. State: Replica: 0731f978074042b0a15a0370c96fdb67, State: Running, Role: LEADER
I20260811 12:36:29.081796 25554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } }
I20260811 12:36:29.082228 25545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:29.090034 25556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0731f978074042b0a15a0370c96fdb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } } }
I20260811 12:36:29.090529 25556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:29.090714 25558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0731f978074042b0a15a0370c96fdb67. Latest consensus state: current_term: 1 leader_uuid: "0731f978074042b0a15a0370c96fdb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } } }
I20260811 12:36:29.091064 25558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:29.105348 25564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:36:29.112020 25564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:36:29.131109 25564 catalog_manager.cc:1347] Generated new cluster ID: 65a77921ba1a4cd7827fb0a51853583d
I20260811 12:36:29.131273 25564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:36:29.141187 25545 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 12:36:29.177917 25564 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:36:29.179615 25564 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:36:29.191668 25564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Generated new TSK 0
I20260811 12:36:29.192952 25564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:36:29.971866 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:46261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45567
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:36:30 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786451790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 12:36:30.474537 25623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:30.474956 25623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:30.475139 25623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:30.486828 25623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:30.487011 25623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:30.487213 25623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:36:30.503404 25623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.204.84:46261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:30.505525 25623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:30.507496 25623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:30.526890 25645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:30.527835 25646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:30.532464 25650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:30.533478 25623 server_base.cc:1034] running on GCE node
I20260811 12:36:30.534395 25623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:30.535799 25623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:30.540294 25623 hybrid_clock.cc:648] HybridClock initialized: now 1786451790540199 us; error 72 us; skew 500 ppm
I20260811 12:36:30.540683 25623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:30 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:30.544034 25623 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:36:30.545854 25623 webserver.cc:460] Webserver started at http://127.3.204.65:38845/ using document root <none> and password file <none>
I20260811 12:36:30.546862 25623 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:30.546993 25623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:30.547309 25623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 12:36:30 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:30.579443 25623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5"
format_stamp: "Formatted at 2026-08-11 12:36:30 on dist-test-slave-lgr5"
server_key: "328dd4fd70717ec2b58661f2d8de2aee"
server_key_iv: "3517e47d2cb87328a3f2342817a89e3f"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:30.581378 25623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5"
format_stamp: "Formatted at 2026-08-11 12:36:30 on dist-test-slave-lgr5"
server_key: "328dd4fd70717ec2b58661f2d8de2aee"
server_key_iv: "3517e47d2cb87328a3f2342817a89e3f"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:30.593528 25623 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.000s
I20260811 12:36:30.625984 25661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:30.627820 25623 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.006s sys 0.005s
I20260811 12:36:30.628048 25623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5"
format_stamp: "Formatted at 2026-08-11 12:36:30 on dist-test-slave-lgr5"
server_key: "328dd4fd70717ec2b58661f2d8de2aee"
server_key_iv: "3517e47d2cb87328a3f2342817a89e3f"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:30.628324 25623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:30.646806 25623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:30.652305 25623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:30.652626 25623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:30.653867 25623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:30.657305 25623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:30.657395 25623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:30.657552 25623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:30.657637 25623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:36:30 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:30.821214 25623 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:39399
I20260811 12:36:30.821282 25859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:39399 every 8 connection(s)
I20260811 12:36:30.823029 25623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:36:30.829421 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 25623
I20260811 12:36:30.829639 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:36:30.830439 3889 external_mini_cluster.cc:1428] Setting key 18a7fed75a5b54e89fac4bd8f2f400c4
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Aug 11 12:36:30 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:30.881575 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:30.827506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:54361 (local address 127.3.204.126:45567)
0811 12:36:30.852789 (+ 25283us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:30.852796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:30.856602 (+ 3806us) server_negotiation.cc:407] Connection header received
0811 12:36:30.856683 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:30.856688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:30.856797 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:30.856897 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:30.860069 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:30.861215 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:30.863654 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:30.864272 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:30.864372 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:30.873383 (+ 9011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:30.873391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:30.873397 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:30.873447 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:30.877514 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:30.878586 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:30.878592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:30.878596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:30.878758 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:30.879328 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:30.879333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:30.879337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:30.879695 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:30.879817 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:30.880918 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0811 12:36:30.881183 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25126,"thread_start_us":25290,"threads_started":1}
I20260811 12:36:30.884210 25862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:30.826814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45567 (local address 127.3.204.65:54361)
0811 12:36:30.853544 (+ 26730us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:30.853608 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:30.856058 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:30.858240 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:30.858254 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:30.858899 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:30.859806 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:30.859840 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:30.861458 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:30.861463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:30.863453 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:30.863479 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:30.864761 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:30.864769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:30.865150 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:30.870159 (+ 5009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:30.870197 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:30.872979 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:30.877726 (+ 4747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:30.877737 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:30.877759 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:30.878429 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:30.878910 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:30.878914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:30.878918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:30.879196 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:30.879922 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:30.879931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:30.880451 (+ 520us) client_negotiation.cc:769] Sending connection context
0811 12:36:30.882136 (+ 1685us) client_negotiation.cc:240] Negotiation successful
0811 12:36:30.882737 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26337,"thread_start_us":30083,"threads_started":1}
I20260811 12:36:30.886457 25860 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45567
I20260811 12:36:30.886801 25860 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:30.887814 25860 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
I20260811 12:36:30.890513 25479 ts_manager.cc:194] Registered new tserver with Master: af1b0e15d5ad4ae5a6a787429f77f6b5 (127.3.204.65:39399)
I20260811 12:36:30.892637 25479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:54361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:36:30.916829 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:46261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45567
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:36:31 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
W20260811 12:36:31.381971 25869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:31.382354 25869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:31.382503 25869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:31.390244 25869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:31.390378 25869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:31.390524 25869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:36:31.400879 25869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.204.84:46261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:31.402298 25869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:31.403568 25869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:31.417339 25897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:31.417373 25898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:31.418170 25903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:31.418653 25869 server_base.cc:1034] running on GCE node
I20260811 12:36:31.419425 25869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:31.420756 25869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:31.422042 25869 hybrid_clock.cc:648] HybridClock initialized: now 1786451791421961 us; error 54 us; skew 500 ppm
I20260811 12:36:31.422444 25869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:31 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451791, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:31.426548 25869 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:36:31.428192 25869 webserver.cc:460] Webserver started at http://127.3.204.66:43721/ using document root <none> and password file <none>
I20260811 12:36:31.428948 25869 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:31.429066 25869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:31.429386 25869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 12:36:31 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451791, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:31.447500 25869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "95284e9468b64ddab63c4fb57d8999d1"
format_stamp: "Formatted at 2026-08-11 12:36:31 on dist-test-slave-lgr5"
server_key: "373aa03717308505d3f9729c1cb0895a"
server_key_iv: "d80c8491cc8f741191867b14cfa67476"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:31.448498 25869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "95284e9468b64ddab63c4fb57d8999d1"
format_stamp: "Formatted at 2026-08-11 12:36:31 on dist-test-slave-lgr5"
server_key: "373aa03717308505d3f9729c1cb0895a"
server_key_iv: "d80c8491cc8f741191867b14cfa67476"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:31.454124 25869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:36:31.478724 25915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:31.480894 25869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 12:36:31.481127 25869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "95284e9468b64ddab63c4fb57d8999d1"
format_stamp: "Formatted at 2026-08-11 12:36:31 on dist-test-slave-lgr5"
server_key: "373aa03717308505d3f9729c1cb0895a"
server_key_iv: "d80c8491cc8f741191867b14cfa67476"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:31.481421 25869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:31.511162 25869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:31.515432 25869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:31.515738 25869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:31.516848 25869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:31.518361 25869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:31.518441 25869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:31.518565 25869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:31.518617 25869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:31.555594 25869 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:45303
I20260811 12:36:31.555610 26100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:45303 every 8 connection(s)
I20260811 12:36:31.557044 25869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:36:31.561708 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 25869
I20260811 12:36:31.561889 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:36:31.562501 3889 external_mini_cluster.cc:1428] Setting key 1d108a1d3d1aaf2ff9d358b6369aa370
Aug 11 12:36:31 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451791, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:31.601675 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:31.560873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45567 (local address 127.3.204.66:40391)
0811 12:36:31.576769 (+ 15896us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:31.576835 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:31.579219 (+ 2384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:31.580466 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:31.580479 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:31.581107 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:31.582040 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:31.582073 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:31.583689 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:31.583694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:31.585793 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:31.585813 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:31.587101 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:31.587108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:31.587417 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:31.588807 (+ 1390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:31.588845 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:31.591740 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:31.596166 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:31.596178 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:31.596198 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:31.596877 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:31.597405 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:31.597410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:31.597414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:31.597717 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:31.598554 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:31.598563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:31.599091 (+ 528us) client_negotiation.cc:769] Sending connection context
0811 12:36:31.599467 (+ 376us) client_negotiation.cc:240] Negotiation successful
0811 12:36:31.600100 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15519,"thread_start_us":15274,"threads_started":1}
I20260811 12:36:31.603478 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:31.560175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:40391 (local address 127.3.204.126:45567)
0811 12:36:31.572569 (+ 12394us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:31.572577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:31.577362 (+ 4785us) server_negotiation.cc:407] Connection header received
0811 12:36:31.579653 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:31.579660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:31.579763 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:31.579846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:31.582274 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:31.583420 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:31.585977 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:31.586641 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:31.586749 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:31.592129 (+ 5380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:31.592137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:31.592143 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:31.592190 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:31.595883 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:31.597075 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:31.597081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:31.597085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:31.597247 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:31.597868 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:31.597873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:31.597877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:31.598312 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:31.598470 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:31.602603 (+ 4133us) server_negotiation.cc:299] Negotiation successful
0811 12:36:31.602894 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12223,"thread_start_us":12083,"threads_started":1}
I20260811 12:36:31.604938 26101 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45567
I20260811 12:36:31.605310 26101 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:31.606230 26101 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
I20260811 12:36:31.608011 25479 ts_manager.cc:194] Registered new tserver with Master: 95284e9468b64ddab63c4fb57d8999d1 (127.3.204.66:45303)
I20260811 12:36:31.609194 25479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:40391
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:36:31.651935 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:46261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45567
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:36:31.896673 25860 heartbeater.cc:503] Master 127.3.204.126:45567 was elected leader, sending a full tablet report...
W20260811 12:36:32.038120 26113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:32.038609 26113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:32.038753 26113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:32.046240 26113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:32.046383 26113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:32.046550 26113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:36:32.059214 26113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.204.84:46261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:32.060856 26113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:32.062497 26113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:32.080359 26128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:32.080459 26126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:32.081208 26130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:32.082240 26113 server_base.cc:1034] running on GCE node
I20260811 12:36:32.082973 26113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:32.084033 26113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:32.085304 26113 hybrid_clock.cc:648] HybridClock initialized: now 1786451792085197 us; error 62 us; skew 500 ppm
I20260811 12:36:32.085747 26113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:32 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:32.090778 26113 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:36:32.092830 26113 webserver.cc:460] Webserver started at http://127.3.204.67:43539/ using document root <none> and password file <none>
I20260811 12:36:32.093773 26113 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:32.093968 26113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:32.094372 26113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 12:36:32 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:32.112500 26113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "1b82827b4915411f9f784659ee343fed"
format_stamp: "Formatted at 2026-08-11 12:36:32 on dist-test-slave-lgr5"
server_key: "d57dacd6acafe9fbc0121190d6eecb10"
server_key_iv: "9138d7cc314dc2424719408a50f37572"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:32.113855 26113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "1b82827b4915411f9f784659ee343fed"
format_stamp: "Formatted at 2026-08-11 12:36:32 on dist-test-slave-lgr5"
server_key: "d57dacd6acafe9fbc0121190d6eecb10"
server_key_iv: "9138d7cc314dc2424719408a50f37572"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:32.121621 26113 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260811 12:36:32.139911 26138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:32.141853 26113 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 12:36:32.142102 26113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "1b82827b4915411f9f784659ee343fed"
format_stamp: "Formatted at 2026-08-11 12:36:32 on dist-test-slave-lgr5"
server_key: "d57dacd6acafe9fbc0121190d6eecb10"
server_key_iv: "9138d7cc314dc2424719408a50f37572"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:32.142411 26113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:32.162110 26113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:32.168128 26113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:32.168632 26113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:32.170423 26113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:32.172878 26113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:32.173034 26113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:32.173240 26113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:32.173352 26113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:32.254485 26113 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:37623
I20260811 12:36:32.254549 26322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:37623 every 8 connection(s)
I20260811 12:36:32.256170 26113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:36:32.259156 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 26113
I20260811 12:36:32.259378 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:36:32.260185 3889 external_mini_cluster.cc:1428] Setting key ff5786fc8685c3d1ea383bbafcc4e13a
Aug 11 12:36:32 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:32.286341 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:32.260022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:35063 (local address 127.3.204.126:45567)
0811 12:36:32.260982 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:32.260987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:32.262882 (+ 1895us) server_negotiation.cc:407] Connection header received
0811 12:36:32.265780 (+ 2898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:32.265785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:32.265921 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:32.266049 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:32.268478 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.269392 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.271639 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.272630 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.272771 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:32.278246 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:32.278252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:32.278255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:32.278293 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:32.282191 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:32.283227 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:32.283232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:32.283234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:32.283511 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:32.284159 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:32.284164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:32.284166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:32.284575 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:32.284717 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:32.285675 (+ 958us) server_negotiation.cc:299] Negotiation successful
0811 12:36:32.285915 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":661,"threads_started":1}
I20260811 12:36:32.287599 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:32.261122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45567 (local address 127.3.204.67:35063)
0811 12:36:32.262722 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:32.262772 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:32.265476 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:32.266366 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:32.266376 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:32.267276 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:32.268271 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:32.268296 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.269605 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.269609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:32.271468 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:32.271484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.272972 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.272982 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:32.273362 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:32.274306 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:32.274338 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:32.277921 (+ 3583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:32.282348 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:32.282357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:32.282372 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:32.283066 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:32.283637 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:32.283641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:32.283643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:32.283932 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:32.284709 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:32.284716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:32.285206 (+ 490us) client_negotiation.cc:769] Sending connection context
0811 12:36:32.285505 (+ 299us) client_negotiation.cc:240] Negotiation successful
0811 12:36:32.286051 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"thread_start_us":776,"threads_started":1}
I20260811 12:36:32.289273 26323 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45567
I20260811 12:36:32.289693 26323 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:32.290778 26323 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
I20260811 12:36:32.292734 25479 ts_manager.cc:194] Registered new tserver with Master: 1b82827b4915411f9f784659ee343fed (127.3.204.67:37623)
I20260811 12:36:32.293637 25479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:35063
I20260811 12:36:32.303443 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:36:32.305256 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 25623
Aug 11 12:36:32 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:36:32.329558 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:32.306265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.3.204.65:39399)
0811 12:36:32.307271 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:32.307279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:32.307307 (+ 28us) server_negotiation.cc:407] Connection header received
0811 12:36:32.307368 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:32.307375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:32.307626 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:32.307761 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:32.310362 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.311969 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.314264 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.315028 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.315147 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:32.321502 (+ 6355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:32.321519 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:32.321537 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:32.321581 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:32.325784 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:32.326791 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:32.326800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:32.326802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:32.327062 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:32.327626 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:32.327630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:32.327633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:32.328198 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:32.328577 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:32.328795 (+ 218us) server_negotiation.cc:299] Negotiation successful
0811 12:36:32.329101 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":605,"threads_started":1}
I20260811 12:36:32.331131 25854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:32.417284 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 25623
I20260811 12:36:32.436010 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 25869
Aug 11 12:36:32 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:36:32.453058 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:32.436993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.3.204.66:45303)
0811 12:36:32.437611 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:32.437620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:32.437645 (+ 25us) server_negotiation.cc:407] Connection header received
0811 12:36:32.437705 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:32.437711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:32.437930 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:32.438098 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:32.439426 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.440474 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.441799 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.442350 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.442470 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:32.446957 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:32.446969 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:32.446985 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:32.447021 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:32.450089 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:32.450830 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:32.450838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:32.450841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:32.450982 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:32.451362 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:32.451366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:32.451368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:32.451771 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:32.452116 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:32.452405 (+ 289us) server_negotiation.cc:299] Negotiation successful
0811 12:36:32.452677 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":310,"threads_started":1}
I20260811 12:36:32.454435 26095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:32.536844 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 25869
I20260811 12:36:32.566845 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 26113
Aug 11 12:36:32 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:36:32.583782 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:32.567798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.3.204.67:37623)
0811 12:36:32.568400 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:32.568408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:32.568439 (+ 31us) server_negotiation.cc:407] Connection header received
0811 12:36:32.568502 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:32.568509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:32.568733 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:32.568898 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:32.570292 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.571379 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.573066 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:32.573643 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:32.573767 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:32.578231 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:32.578244 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:32.578262 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:32.578305 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:32.580889 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:32.581584 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:32.581591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:32.581594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:32.581733 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:32.582159 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:32.582163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:32.582165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:32.582550 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:32.582878 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:32.583178 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 12:36:32.583399 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":290,"threads_started":1}
I20260811 12:36:32.585165 26312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:32.667932 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 26113
I20260811 12:36:32.689680 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 25418
I20260811 12:36:32.691066 25536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:32.797592 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 25418
I20260811 12:36:32.829402 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:45567
--webserver_interface=127.3.204.126
--webserver_port=38233
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:45567
--ranger_config_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:46261/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.204.84:46261/kms with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:33.166065 26339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:33.166409 26339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:33.166554 26339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:33.173938 26339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:33.174073 26339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:33.174150 26339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:33.174225 26339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:33.174289 26339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:33.184767 26339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.204.84:46261/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:45567
--ranger_config_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=38233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:33.186118 26339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:33.187453 26339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:33.199095 26355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:33.199162 26365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:33.199186 26358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:33.200222 26339 server_base.cc:1034] running on GCE node
I20260811 12:36:33.201033 26339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:33.202478 26339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:33.203687 26339 hybrid_clock.cc:648] HybridClock initialized: now 1786451793203632 us; error 49 us; skew 500 ppm
I20260811 12:36:33.204051 26339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:33 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451793, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:33.207122 26339 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:36:33.208664 26339 webserver.cc:460] Webserver started at http://127.3.204.126:38233/ using document root <none> and password file <none>
I20260811 12:36:33.209296 26339 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:33.209410 26339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 12:36:33 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451793, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:33.284579 26339 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 12:36:33.287855 26379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:33.289469 26339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:36:33.289677 26339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "0731f978074042b0a15a0370c96fdb67"
format_stamp: "Formatted at 2026-08-11 12:36:28 on dist-test-slave-lgr5"
server_key: "5e4afea9b06ec35620a0b726a8251ee4"
server_key_iv: "a5cfe054d67e51cd8c2a4e09171ae2e3"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:33.290771 26339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:33.314736 26339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:33.318814 26339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:33.319118 26339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:33.339794 26339 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:45567
I20260811 12:36:33.339818 26465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:45567 every 8 connection(s)
I20260811 12:36:33.341480 26339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:36:33.349167 26467 sys_catalog.cc:263] Verifying existing consensus state
I20260811 12:36:33.349367 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 26339
I20260811 12:36:33.350453 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:46261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:39399
--local_ip_for_outbound_sockets=127.3.204.65
--tserver_master_addrs=127.3.204.126:45567
--webserver_port=38845
--webserver_interface=127.3.204.65
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.204.84:46261/kms with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:36:33.354008 26467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Bootstrap starting.
I20260811 12:36:33.374714 26467 log.cc:824] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:33.383911 26467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:36:33.384507 26467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Bootstrap complete.
I20260811 12:36:33.394819 26467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } }
I20260811 12:36:33.395900 26467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0731f978074042b0a15a0370c96fdb67, State: Initialized, Role: FOLLOWER
I20260811 12:36:33.396647 26467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } }
I20260811 12:36:33.396880 26467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:36:33.397003 26467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:36:33.397163 26467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 1 FOLLOWER]: Advancing to term 2
I20260811 12:36:33.401219 26467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } }
I20260811 12:36:33.401734 26467 leader_election.cc:302] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0731f978074042b0a15a0370c96fdb67; no voters:
I20260811 12:36:33.402376 26467 leader_election.cc:288] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 12:36:33.402576 26477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 12:36:33.403618 26477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [term 2 LEADER]: Becoming Leader. State: Replica: 0731f978074042b0a15a0370c96fdb67, State: Running, Role: LEADER
I20260811 12:36:33.404217 26477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } }
I20260811 12:36:33.404631 26467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:33.406850 26480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0731f978074042b0a15a0370c96fdb67. Latest consensus state: current_term: 2 leader_uuid: "0731f978074042b0a15a0370c96fdb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } } }
I20260811 12:36:33.407821 26480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:33.407439 26479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0731f978074042b0a15a0370c96fdb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731f978074042b0a15a0370c96fdb67" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 45567 } } }
I20260811 12:36:33.408322 26479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:33.419433 26484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:36:33.425555 26484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:36:33.426956 26484 catalog_manager.cc:1259] Loaded cluster ID: 65a77921ba1a4cd7827fb0a51853583d
I20260811 12:36:33.427137 26484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:36:33.431499 26484 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:36:33.433177 26484 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Loaded TSK: 0
I20260811 12:36:33.434913 26484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:36:33.445354 26467 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260811 12:36:33.822506 26469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:33.822945 26469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:33.823105 26469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:33.834646 26469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:33.834810 26469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:33.835029 26469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:36:33.850803 26469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.204.84:46261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:39399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=38845
--tserver_master_addrs=127.3.204.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:33.852698 26469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:33.854511 26469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:33.889782 26527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:33.890429 26528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:33.897809 26532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:33.900249 26469 server_base.cc:1034] running on GCE node
I20260811 12:36:33.901170 26469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:33.909808 26469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:33.911088 26469 hybrid_clock.cc:648] HybridClock initialized: now 1786451793911016 us; error 67 us; skew 500 ppm
I20260811 12:36:33.911531 26469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:33 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451793, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:33.923887 26469 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:36:33.933947 26469 webserver.cc:460] Webserver started at http://127.3.204.65:38845/ using document root <none> and password file <none>
I20260811 12:36:33.935184 26469 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:33.935474 26469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 12:36:33 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451793, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:33.987524 26469 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.006s sys 0.011s
I20260811 12:36:34.000892 26547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:34.003508 26469 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.000s sys 0.012s
I20260811 12:36:34.003774 26469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5"
format_stamp: "Formatted at 2026-08-11 12:36:30 on dist-test-slave-lgr5"
server_key: "328dd4fd70717ec2b58661f2d8de2aee"
server_key_iv: "3517e47d2cb87328a3f2342817a89e3f"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:34.005395 26469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:34.036619 26469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:34.042351 26469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:34.042714 26469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:34.046392 26469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:34.048931 26469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:34.049049 26469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:34.049166 26469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:34.049247 26469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:34.116460 26469 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:39399
I20260811 12:36:34.116489 26741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:39399 every 8 connection(s)
I20260811 12:36:34.118099 26469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:36:34.124377 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 26469
I20260811 12:36:34.125213 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:46261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:45303
--local_ip_for_outbound_sockets=127.3.204.66
--tserver_master_addrs=127.3.204.126:45567
--webserver_port=43721
--webserver_interface=127.3.204.66
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.204.84:46261/kms with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:36:34 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451793, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:34.154350 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:34.122085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45567 (local address 127.3.204.65:48551)
0811 12:36:34.124532 (+ 2447us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:34.124595 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:34.126962 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:34.130414 (+ 3452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:34.130426 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:34.131095 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:34.131931 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:34.131959 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:34.134555 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:34.134561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:34.136737 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:34.136774 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:34.138745 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:34.138755 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:34.139151 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:34.141719 (+ 2568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:34.141751 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:34.144436 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:34.148872 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:34.148881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:34.148909 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:34.149553 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:34.150134 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:34.150138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:34.150141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:34.150386 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:34.151538 (+ 1152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:34.151545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:34.151969 (+ 424us) client_negotiation.cc:769] Sending connection context
0811 12:36:34.152287 (+ 318us) client_negotiation.cc:240] Negotiation successful
0811 12:36:34.152859 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2107,"thread_start_us":4818,"threads_started":1}
I20260811 12:36:34.157374 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:34.123356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:48551 (local address 127.3.204.126:45567)
0811 12:36:34.128196 (+ 4840us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:34.128215 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:34.128265 (+ 50us) server_negotiation.cc:407] Connection header received
0811 12:36:34.128746 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:34.128767 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:34.129236 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:34.129673 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:34.132237 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:34.134309 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:34.136962 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:34.138062 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:34.138290 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:34.144820 (+ 6530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:34.144832 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:34.144855 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:34.144919 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:34.148665 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:34.149732 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:34.149743 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:34.149751 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:34.149981 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:34.150531 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:34.150536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:34.150539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:34.151251 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:34.151444 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:34.155363 (+ 3919us) server_negotiation.cc:299] Negotiation successful
0811 12:36:34.155878 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4532,"thread_start_us":4639,"threads_started":1}
I20260811 12:36:34.162441 26742 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45567
I20260811 12:36:34.162774 26742 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:34.163666 26742 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
W20260811 12:36:34.167339 26742 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:45567 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 12:36:34.167586 26742 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:34.167884 26742 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
I20260811 12:36:34.169379 26742 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:34.169771 26742 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
W20260811 12:36:34.170970 26742 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 12:36:34.534722 26746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:34.535171 26746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:34.535336 26746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:34.547046 26746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:34.547202 26746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:34.547394 26746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:36:34.562875 26746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.204.84:46261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:45303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=43721
--tserver_master_addrs=127.3.204.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:34.564769 26746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:34.566567 26746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:34.592418 26780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:34.592451 26779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:34.597678 26782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:34.598654 26746 server_base.cc:1034] running on GCE node
I20260811 12:36:34.599589 26746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:34.601042 26746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:34.602308 26746 hybrid_clock.cc:648] HybridClock initialized: now 1786451794602235 us; error 58 us; skew 500 ppm
I20260811 12:36:34.602730 26746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:34 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:34.607280 26746 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:36:34.609535 26746 webserver.cc:460] Webserver started at http://127.3.204.66:43721/ using document root <none> and password file <none>
I20260811 12:36:34.610558 26746 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:34.610726 26746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 12:36:34 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
Aug 11 12:36:34 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786451794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:34.654443 26746 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.010s sys 0.003s
I20260811 12:36:34.672806 26795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:34.675355 26746 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.005s sys 0.012s
I20260811 12:36:34.675583 26746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "95284e9468b64ddab63c4fb57d8999d1"
format_stamp: "Formatted at 2026-08-11 12:36:31 on dist-test-slave-lgr5"
server_key: "373aa03717308505d3f9729c1cb0895a"
server_key_iv: "d80c8491cc8f741191867b14cfa67476"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:34.677201 26746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:34.729177 26746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:34.733603 26746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:34.733893 26746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:34.735018 26746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:34.736656 26746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:34.736756 26746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:34.736869 26746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:34.736950 26746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:34.786355 26746 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:45303
I20260811 12:36:34.786405 26985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:45303 every 8 connection(s)
I20260811 12:36:34.787962 26746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:36:34.788954 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 26746
I20260811 12:36:34.789791 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.204.84:46261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:37623
--local_ip_for_outbound_sockets=127.3.204.67
--tserver_master_addrs=127.3.204.126:45567
--webserver_port=43539
--webserver_interface=127.3.204.67
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.204.84:46261/kms with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:36:34 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:34.867528 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:34.796290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:53341 (local address 127.3.204.126:45567)
0811 12:36:34.832500 (+ 36210us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:34.832508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:34.832539 (+ 31us) server_negotiation.cc:407] Connection header received
0811 12:36:34.841281 (+ 8742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:34.841288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:34.841422 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:34.841516 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:34.849629 (+ 8113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:34.850776 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:34.853299 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:34.853944 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:34.854059 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:34.859677 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:34.859686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:34.859691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:34.859739 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:34.863339 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:34.864515 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:34.864522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:34.864526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:34.864705 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:34.865302 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:34.865308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:34.865311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:34.865717 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:34.865853 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:34.866872 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0811 12:36:34.867131 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36047,"thread_start_us":43803,"threads_started":1}
I20260811 12:36:34.869833 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:34.795610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45567 (local address 127.3.204.66:53341)
0811 12:36:34.829626 (+ 34016us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:34.829701 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:34.840769 (+ 11068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:34.842240 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:34.842255 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:34.843062 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:34.844054 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:34.844112 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:34.851081 (+ 6969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:34.851087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:34.853081 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:34.853102 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:34.854502 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:34.854511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:34.854790 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:34.856459 (+ 1669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:34.856508 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:34.859315 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:34.863588 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:34.863597 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:34.863613 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:34.864324 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:34.864875 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:34.864881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:34.864884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:34.865161 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:34.865958 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:34.865966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:34.866388 (+ 422us) client_negotiation.cc:769] Sending connection context
0811 12:36:34.867785 (+ 1397us) client_negotiation.cc:240] Negotiation successful
0811 12:36:34.868411 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33556,"thread_start_us":36825,"threads_started":1}
I20260811 12:36:34.871673 26988 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45567
I20260811 12:36:34.872053 26988 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:34.873090 26988 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
I20260811 12:36:34.875425 26411 ts_manager.cc:194] Registered new tserver with Master: 95284e9468b64ddab63c4fb57d8999d1 (127.3.204.66:45303)
I20260811 12:36:34.877823 26411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:53341
Aug 11 12:36:35 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:35.171120 26742 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:35.171581 26742 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
I20260811 12:36:35.172888 26411 ts_manager.cc:194] Registered new tserver with Master: af1b0e15d5ad4ae5a6a787429f77f6b5 (127.3.204.65:39399)
I20260811 12:36:35.174078 26411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:48551
W20260811 12:36:35.240296 26992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:35.240607 26992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:35.240773 26992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:35.248590 26992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:35.248723 26992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:35.248875 26992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:36:35.262173 26992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.204.84:46261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:37623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=43539
--tserver_master_addrs=127.3.204.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:35.264183 26992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:35.266258 26992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:35.284446 27001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:35.284684 27002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:35.292117 27005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:35.293078 26992 server_base.cc:1034] running on GCE node
I20260811 12:36:35.293936 26992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:35.295301 26992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:35.296583 26992 hybrid_clock.cc:648] HybridClock initialized: now 1786451795296490 us; error 77 us; skew 500 ppm
I20260811 12:36:35.296984 26992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:35 dist-test-slave-lgr5 krb5kdc[24624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:35.301270 26992 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:36:35.303375 26992 webserver.cc:460] Webserver started at http://127.3.204.67:43539/ using document root <none> and password file <none>
I20260811 12:36:35.304417 26992 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:35.304589 26992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 12:36:35 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:35.355125 26992 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.006s sys 0.015s
I20260811 12:36:35.359282 27024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:35.361327 26992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 12:36:35.361549 26992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "1b82827b4915411f9f784659ee343fed"
format_stamp: "Formatted at 2026-08-11 12:36:32 on dist-test-slave-lgr5"
server_key: "d57dacd6acafe9fbc0121190d6eecb10"
server_key_iv: "9138d7cc314dc2424719408a50f37572"
server_key_version: "kuduclusterkey@0"
I20260811 12:36:35.363130 26992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:35.393692 26992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:35.399544 26992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:35.399865 26992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:35.401254 26992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:35.403152 26992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:35.403270 26992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:35.403385 26992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:35.403466 26992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 12:36:35 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for HTTP/127.3.204.84@KRBTEST.COM
I20260811 12:36:35.635902 26992 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:37623
I20260811 12:36:35.635960 27224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:37623 every 8 connection(s)
I20260811 12:36:35.638039 26992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:36:35.641674 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 26992
I20260811 12:36:35.673285 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.643237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.3.204.126:45567)
0811 12:36:35.654897 (+ 11660us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:35.654906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:35.654936 (+ 30us) server_negotiation.cc:407] Connection header received
0811 12:36:35.655068 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:35.655075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:35.655173 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:35.655322 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:35.659074 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.659832 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.661770 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.662280 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.662397 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.665968 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:35.665974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:35.665977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:35.666022 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:35.668763 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:35.670021 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:35.670027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:35.670031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:35.670206 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:35.670720 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:35.670725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:35.670728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:35.671080 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:35.671340 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:35.672664 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0811 12:36:35.672892 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11504,"thread_start_us":13906,"threads_started":1}
Aug 11 12:36:35 dist-test-slave-lgr5 krb5kdc[24624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:35.682360 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.649306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:44715 (local address 127.3.204.126:45567)
0811 12:36:35.657101 (+ 7795us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:35.657108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:35.657135 (+ 27us) server_negotiation.cc:407] Connection header received
0811 12:36:35.660128 (+ 2993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:35.660134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:35.660230 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:35.660330 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:35.663582 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.664813 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.670065 (+ 5252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.670701 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.670816 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.675396 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:35.675407 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:35.675417 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:35.675569 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:35.678766 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:35.679555 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:35.679562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:35.679565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:35.679690 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:35.680202 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:35.680205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:35.680207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:35.680483 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:35.680608 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:35.681385 (+ 777us) server_negotiation.cc:299] Negotiation successful
0811 12:36:35.681998 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7663,"thread_start_us":5323,"threads_started":1}
I20260811 12:36:35.683207 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.648722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:45567 (local address 127.3.204.67:44715)
0811 12:36:35.656181 (+ 7459us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:35.656233 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:35.658642 (+ 2409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:35.660856 (+ 2214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:35.660870 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:35.661516 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:35.663303 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:35.663337 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.665013 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.665019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:35.669840 (+ 4821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:35.669854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.671169 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.671178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:35.671500 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.672276 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:35.672308 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:35.674892 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:35.678933 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:35.678943 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:35.678960 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:35.679420 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:35.679806 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:35.679810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:35.679812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:35.680046 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:35.680624 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:35.680632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:35.680994 (+ 362us) client_negotiation.cc:769] Sending connection context
0811 12:36:35.681246 (+ 252us) client_negotiation.cc:240] Negotiation successful
0811 12:36:35.681863 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7081,"thread_start_us":7100,"threads_started":1}
I20260811 12:36:35.684818 27229 heartbeater.cc:347] Connected to a master server at 127.3.204.126:45567
I20260811 12:36:35.685189 27229 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:35.686079 27229 heartbeater.cc:511] Master 127.3.204.126:45567 requested a full tablet report, sending...
I20260811 12:36:35.687863 26411 ts_manager.cc:194] Registered new tserver with Master: 1b82827b4915411f9f784659ee343fed (127.3.204.67:37623)
I20260811 12:36:35.689045 26411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:44715
I20260811 12:36:35.694499 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:36:35.713594 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.697938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.3.204.126:45567)
0811 12:36:35.698120 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:35.698127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:35.698830 (+ 703us) server_negotiation.cc:407] Connection header received
0811 12:36:35.699008 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:35.699014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:35.699087 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:35.699167 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:35.700778 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.701489 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.702855 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.703278 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.703388 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.706882 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:35.706889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:35.706894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:35.706938 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:35.710464 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:35.711479 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:35.711484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:35.711487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:35.711645 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:35.712114 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:35.712121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:35.712125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:35.712438 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:35.712754 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:35.713043 (+ 289us) server_negotiation.cc:299] Negotiation successful
0811 12:36:35.713265 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 12:36:35.719587 26411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:36:35.730306 26411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:36:35.796938 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.755934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:45303 (local address 127.0.0.1:33906)
0811 12:36:35.763349 (+ 7415us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:35.763367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:35.763480 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:35.768776 (+ 5296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:35.768781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:35.768818 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:35.769232 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:35.769245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.772684 (+ 3439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.772691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:35.774877 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:35.774899 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.795839 (+ 20940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.795848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:35.796098 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.796141 (+ 43us) client_negotiation.cc:769] Sending connection context
0811 12:36:35.796237 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 12:36:35.796519 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7297,"thread_start_us":7180,"threads_started":1}
I20260811 12:36:35.798122 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.757253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.3.204.66:45303)
0811 12:36:35.757838 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:35.757845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:35.763614 (+ 5769us) server_negotiation.cc:407] Connection header received
0811 12:36:35.763735 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:35.763742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:35.764001 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:35.764217 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:35.769475 (+ 5258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.771216 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.793058 (+ 21842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.794959 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.795138 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.795568 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:35.797538 (+ 1970us) server_negotiation.cc:299] Negotiation successful
0811 12:36:35.797756 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":361,"threads_started":1}
I20260811 12:36:35.799441 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.756828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.3.204.67:37623)
0811 12:36:35.757494 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:35.757504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:35.760864 (+ 3360us) server_negotiation.cc:407] Connection header received
0811 12:36:35.761296 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:35.761305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:35.761645 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:35.761815 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:35.764472 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.766982 (+ 2510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.789823 (+ 22841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.791589 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.791734 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.792021 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:35.798803 (+ 6782us) server_negotiation.cc:299] Negotiation successful
0811 12:36:35.799058 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":991,"threads_started":1}
I20260811 12:36:35.800590 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.755827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:37623 (local address 127.0.0.1:36790)
0811 12:36:35.760693 (+ 4866us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:35.760742 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:35.761035 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:35.762267 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:35.762275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:35.762312 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:35.762821 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:35.762835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.767204 (+ 4369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.767210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:35.788862 (+ 21652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:35.788900 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.792348 (+ 3448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.792356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:35.792653 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.792715 (+ 62us) client_negotiation.cc:769] Sending connection context
0811 12:36:35.799930 (+ 7215us) client_negotiation.cc:240] Negotiation successful
0811 12:36:35.800184 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4601,"thread_start_us":4472,"threads_started":1}
I20260811 12:36:35.805320 26870 tablet_service.cc:1467] Processing CreateTablet for tablet 055e9ab0fd63490681d8dec27711b05a (DEFAULT_TABLE table=test-table [id=418b4dc512ef42c49dceb1a7c005ad6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:35.807097 26870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055e9ab0fd63490681d8dec27711b05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:35.809904 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.763423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:39399 (local address 127.0.0.1:44224)
0811 12:36:35.801448 (+ 38025us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:35.801470 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:35.801616 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:35.802472 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:35.802478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:35.802518 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:35.802721 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:35.802733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.804898 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.804904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:35.806877 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:35.806896 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.809090 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.809098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:35.809286 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.809326 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 12:36:35.809428 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 12:36:35.809600 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37904,"thread_start_us":46016,"threads_started":1}
I20260811 12:36:35.810654 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:35.761649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.3.204.65:39399)
0811 12:36:35.789639 (+ 27990us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:35.789649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:35.801783 (+ 12134us) server_negotiation.cc:407] Connection header received
0811 12:36:35.801917 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:35.801924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:35.802235 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:35.802445 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:35.802947 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.804674 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.807089 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:35.808577 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:35.808721 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:35.808973 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:35.810114 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0811 12:36:35.810290 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27786,"thread_start_us":27607,"threads_started":1}
I20260811 12:36:35.813828 27105 tablet_service.cc:1467] Processing CreateTablet for tablet 055e9ab0fd63490681d8dec27711b05a (DEFAULT_TABLE table=test-table [id=418b4dc512ef42c49dceb1a7c005ad6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:35.814683 26620 tablet_service.cc:1467] Processing CreateTablet for tablet 055e9ab0fd63490681d8dec27711b05a (DEFAULT_TABLE table=test-table [id=418b4dc512ef42c49dceb1a7c005ad6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:35.815689 27105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055e9ab0fd63490681d8dec27711b05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:35.816181 26620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055e9ab0fd63490681d8dec27711b05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:35.833649 27276 tablet_bootstrap.cc:492] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Bootstrap starting.
I20260811 12:36:35.836951 27274 tablet_bootstrap.cc:492] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Bootstrap starting.
I20260811 12:36:35.840747 27276 tablet_bootstrap.cc:654] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:35.842365 27276 log.cc:824] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:35.843161 27274 tablet_bootstrap.cc:654] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:35.846594 27275 tablet_bootstrap.cc:492] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Bootstrap starting.
I20260811 12:36:35.848728 27274 log.cc:824] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:35.850566 27275 tablet_bootstrap.cc:654] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:35.851977 27275 log.cc:824] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:35.852587 27276 tablet_bootstrap.cc:492] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: No bootstrap required, opened a new log
I20260811 12:36:35.852988 27276 ts_tablet_manager.cc:1397] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20260811 12:36:35.863529 27275 tablet_bootstrap.cc:492] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: No bootstrap required, opened a new log
I20260811 12:36:35.863883 27275 ts_tablet_manager.cc:1397] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.013s
I20260811 12:36:35.866609 27274 tablet_bootstrap.cc:492] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: No bootstrap required, opened a new log
I20260811 12:36:35.866886 27274 ts_tablet_manager.cc:1397] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.009s
I20260811 12:36:35.869431 27276 raft_consensus.cc:348] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:35.870067 27276 raft_consensus.cc:374] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:35.870217 27276 raft_consensus.cc:729] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b82827b4915411f9f784659ee343fed, State: Initialized, Role: FOLLOWER
I20260811 12:36:35.871031 27276 consensus_queue.cc:260] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:35.875554 27275 raft_consensus.cc:348] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:35.876111 27275 raft_consensus.cc:374] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:35.876257 27275 raft_consensus.cc:729] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1b0e15d5ad4ae5a6a787429f77f6b5, State: Initialized, Role: FOLLOWER
I20260811 12:36:35.877043 27275 consensus_queue.cc:260] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:35.877439 27274 raft_consensus.cc:348] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:35.877995 27274 raft_consensus.cc:374] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:35.878155 27274 raft_consensus.cc:729] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95284e9468b64ddab63c4fb57d8999d1, State: Initialized, Role: FOLLOWER
I20260811 12:36:35.878582 27229 heartbeater.cc:503] Master 127.3.204.126:45567 was elected leader, sending a full tablet report...
I20260811 12:36:35.878908 27274 consensus_queue.cc:260] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:35.879787 27276 ts_tablet_manager.cc:1428] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Time spent starting tablet: real 0.027s user 0.015s sys 0.005s
I20260811 12:36:35.882627 26742 heartbeater.cc:503] Master 127.3.204.126:45567 was elected leader, sending a full tablet report...
I20260811 12:36:35.883535 27275 ts_tablet_manager.cc:1428] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Time spent starting tablet: real 0.020s user 0.010s sys 0.008s
I20260811 12:36:35.884208 26988 heartbeater.cc:503] Master 127.3.204.126:45567 was elected leader, sending a full tablet report...
I20260811 12:36:35.889909 27274 ts_tablet_manager.cc:1428] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Time spent starting tablet: real 0.023s user 0.015s sys 0.005s
W20260811 12:36:35.906219 27236 tablet.cc:2369] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:36:36.062794 27285 raft_consensus.cc:482] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:36:36.063161 27285 raft_consensus.cc:504] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:36.065357 27285 leader_election.cc:288] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b82827b4915411f9f784659ee343fed (127.3.204.67:37623), 95284e9468b64ddab63c4fb57d8999d1 (127.3.204.66:45303)
I20260811 12:36:36.078495 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:36.065871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:57521 (local address 127.3.204.67:37623)
0811 12:36:36.066128 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:36.066135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:36.068235 (+ 2100us) server_negotiation.cc:407] Connection header received
0811 12:36:36.068629 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:36.068634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:36.068707 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:36.068806 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:36.070300 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.071510 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.075329 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.077019 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.077277 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:36.077379 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:36.077915 (+ 536us) server_negotiation.cc:299] Negotiation successful
0811 12:36:36.078080 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 12:36:36.078707 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:36.065766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:37623 (local address 127.3.204.65:57521)
0811 12:36:36.068043 (+ 2277us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:36.068098 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:36.068436 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:36.068889 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:36.068895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:36.068924 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:36.070106 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:36.070130 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.071657 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.071663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:36.075089 (+ 3426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:36.075108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.077392 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.077400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:36.077671 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:36.077736 (+ 65us) client_negotiation.cc:769] Sending connection context
0811 12:36:36.077845 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 12:36:36.077993 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2054,"mutex_wait_us":29,"thread_start_us":1607,"threads_started":1}
I20260811 12:36:36.078799 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:36.066270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:55309 (local address 127.3.204.66:45303)
0811 12:36:36.066563 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:36.066570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:36.068781 (+ 2211us) server_negotiation.cc:407] Connection header received
0811 12:36:36.069149 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:36.069157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:36.069260 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:36.069381 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:36.070761 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.071958 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.075225 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.077129 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.077318 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:36.077398 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:36.078302 (+ 904us) server_negotiation.cc:299] Negotiation successful
0811 12:36:36.078452 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 12:36:36.079859 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:36.066755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:45303 (local address 127.3.204.65:55309)
0811 12:36:36.068639 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:36.068657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:36.068914 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:36.069399 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:36.069404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:36.069429 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:36.070522 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:36.070537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.072172 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.072177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:36.074937 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:36.074958 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.077672 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.077678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:36.078065 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:36.078105 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 12:36:36.079070 (+ 965us) client_negotiation.cc:240] Negotiation successful
0811 12:36:36.079226 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"mutex_wait_us":84,"thread_start_us":1367,"threads_started":1}
I20260811 12:36:36.080677 27143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055e9ab0fd63490681d8dec27711b05a" candidate_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b82827b4915411f9f784659ee343fed" is_pre_election: true
I20260811 12:36:36.081480 27143 raft_consensus.cc:2454] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af1b0e15d5ad4ae5a6a787429f77f6b5 in term 0.
I20260811 12:36:36.081634 26903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055e9ab0fd63490681d8dec27711b05a" candidate_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95284e9468b64ddab63c4fb57d8999d1" is_pre_election: true
I20260811 12:36:36.082489 26548 leader_election.cc:302] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b82827b4915411f9f784659ee343fed, af1b0e15d5ad4ae5a6a787429f77f6b5; no voters:
I20260811 12:36:36.082612 26903 raft_consensus.cc:2454] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af1b0e15d5ad4ae5a6a787429f77f6b5 in term 0.
I20260811 12:36:36.083184 27285 raft_consensus.cc:2781] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:36:36.083433 27285 raft_consensus.cc:482] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:36:36.083611 27285 raft_consensus.cc:3037] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:36.087251 27285 raft_consensus.cc:504] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:36.088114 27285 leader_election.cc:288] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [CANDIDATE]: Term 1 election: Requested vote from peers 1b82827b4915411f9f784659ee343fed (127.3.204.67:37623), 95284e9468b64ddab63c4fb57d8999d1 (127.3.204.66:45303)
I20260811 12:36:36.088755 26903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055e9ab0fd63490681d8dec27711b05a" candidate_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95284e9468b64ddab63c4fb57d8999d1"
I20260811 12:36:36.088763 27143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055e9ab0fd63490681d8dec27711b05a" candidate_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b82827b4915411f9f784659ee343fed"
I20260811 12:36:36.088979 26903 raft_consensus.cc:3037] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:36.089067 27143 raft_consensus.cc:3037] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:36.092947 26903 raft_consensus.cc:2454] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af1b0e15d5ad4ae5a6a787429f77f6b5 in term 1.
I20260811 12:36:36.093431 27143 raft_consensus.cc:2454] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af1b0e15d5ad4ae5a6a787429f77f6b5 in term 1.
I20260811 12:36:36.093565 26550 leader_election.cc:302] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95284e9468b64ddab63c4fb57d8999d1, af1b0e15d5ad4ae5a6a787429f77f6b5; no voters:
I20260811 12:36:36.094097 27285 raft_consensus.cc:2781] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:36:36.095127 27285 raft_consensus.cc:686] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 1 LEADER]: Becoming Leader. State: Replica: af1b0e15d5ad4ae5a6a787429f77f6b5, State: Running, Role: LEADER
W20260811 12:36:36.096105 26989 tablet.cc:2369] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:36:36.096361 27285 consensus_queue.cc:237] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } }
I20260811 12:36:36.102867 26410 catalog_manager.cc:5495] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to af1b0e15d5ad4ae5a6a787429f77f6b5 (127.3.204.65). New cstate: current_term: 1 leader_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af1b0e15d5ad4ae5a6a787429f77f6b5" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 39399 } health_report { overall_health: HEALTHY } } }
I20260811 12:36:36.126173 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:36.118777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.3.204.65:39399)
0811 12:36:36.119057 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:36.119063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:36.119281 (+ 218us) server_negotiation.cc:407] Connection header received
0811 12:36:36.119543 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:36.119547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:36.119623 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:36.119756 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:36:36.120338 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.121433 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.123288 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.123986 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.124126 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:36.124861 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:36:36.125414 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:36:36.125590 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:36.125640 (+ 50us) server_negotiation.cc:299] Negotiation successful
0811 12:36:36.125729 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260811 12:36:36.130219 26743 tablet.cc:2369] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:36:36.137907 26903 raft_consensus.cc:1261] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 1 FOLLOWER]: Refusing update from remote peer af1b0e15d5ad4ae5a6a787429f77f6b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:36:36.137954 27143 raft_consensus.cc:1261] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 1 FOLLOWER]: Refusing update from remote peer af1b0e15d5ad4ae5a6a787429f77f6b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:36:36.139210 27285 consensus_queue.cc:1035] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b82827b4915411f9f784659ee343fed" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:36:36.139665 27290 consensus_queue.cc:1035] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95284e9468b64ddab63c4fb57d8999d1" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:36:36.154304 27295 mvcc.cc:205] Tried to move back new op lower bound from 7317306556967698432 to 7317306556815876096. Current Snapshot: MvccSnapshot[applied={T|T < 7317306556967698432}]
I20260811 12:36:36.174183 26410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:36:36.174777 26410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:36:36.179883 26410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0731f978074042b0a15a0370c96fdb67: Sending DeleteTablet for 3 replicas of tablet 055e9ab0fd63490681d8dec27711b05a
I20260811 12:36:36.181243 27105 tablet_service.cc:1514] Processing DeleteTablet for tablet 055e9ab0fd63490681d8dec27711b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:36 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:36790
I20260811 12:36:36.181331 26870 tablet_service.cc:1514] Processing DeleteTablet for tablet 055e9ab0fd63490681d8dec27711b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:36 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:33906
I20260811 12:36:36.181732 26620 tablet_service.cc:1514] Processing DeleteTablet for tablet 055e9ab0fd63490681d8dec27711b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:36 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:44224
I20260811 12:36:36.182914 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 26469
I20260811 12:36:36.183122 27314 tablet_replica.cc:330] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: stopping tablet replica
I20260811 12:36:36.183626 27314 raft_consensus.cc:2229] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:36:36.187126 27313 tablet_replica.cc:330] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: stopping tablet replica
I20260811 12:36:36.187755 27317 tablet_replica.cc:330] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: stopping tablet replica
I20260811 12:36:36.188316 27317 raft_consensus.cc:2229] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:36:36.189182 27313 raft_consensus.cc:2229] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:36:36.189235 27317 raft_consensus.cc:2258] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:36.189947 27313 raft_consensus.cc:2258] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:36.191591 27314 raft_consensus.cc:2258] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:36.192155 27317 ts_tablet_manager.cc:1905] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:36.193923 27313 ts_tablet_manager.cc:1905] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:36.194067 27314 ts_tablet_manager.cc:1905] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:36.203372 27313 ts_tablet_manager.cc:1918] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:36.203372 27314 ts_tablet_manager.cc:1918] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:36.203617 27314 log.cc:1196] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/055e9ab0fd63490681d8dec27711b05a
I20260811 12:36:36.203877 27313 log.cc:1196] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/055e9ab0fd63490681d8dec27711b05a
I20260811 12:36:36.204670 27314 ts_tablet_manager.cc:1939] T 055e9ab0fd63490681d8dec27711b05a P 95284e9468b64ddab63c4fb57d8999d1: Deleting consensus metadata
I20260811 12:36:36.204909 27313 ts_tablet_manager.cc:1939] T 055e9ab0fd63490681d8dec27711b05a P 1b82827b4915411f9f784659ee343fed: Deleting consensus metadata
I20260811 12:36:36.207204 26380 catalog_manager.cc:4841] TS 1b82827b4915411f9f784659ee343fed (127.3.204.67:37623): tablet 055e9ab0fd63490681d8dec27711b05a (table test-table [id=418b4dc512ef42c49dceb1a7c005ad6f]) successfully deleted
I20260811 12:36:36.207904 26383 catalog_manager.cc:4841] TS 95284e9468b64ddab63c4fb57d8999d1 (127.3.204.66:45303): tablet 055e9ab0fd63490681d8dec27711b05a (table test-table [id=418b4dc512ef42c49dceb1a7c005ad6f]) successfully deleted
I20260811 12:36:36.208268 27317 ts_tablet_manager.cc:1918] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:36.208570 27317 log.cc:1196] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/055e9ab0fd63490681d8dec27711b05a
I20260811 12:36:36.209333 27317 ts_tablet_manager.cc:1939] T 055e9ab0fd63490681d8dec27711b05a P af1b0e15d5ad4ae5a6a787429f77f6b5: Deleting consensus metadata
I20260811 12:36:36.211287 26380 catalog_manager.cc:4841] TS af1b0e15d5ad4ae5a6a787429f77f6b5 (127.3.204.65:39399): tablet 055e9ab0fd63490681d8dec27711b05a (table test-table [id=418b4dc512ef42c49dceb1a7c005ad6f]) successfully deleted
I20260811 12:36:36.220973 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:36.187343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.3.204.65:39399)
0811 12:36:36.187542 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:36.187550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:36.191648 (+ 4098us) server_negotiation.cc:407] Connection header received
0811 12:36:36.199187 (+ 7539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:36.199194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:36.199415 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:36.199557 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:36.202367 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.204564 (+ 2197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.206833 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.207681 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.207791 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:36.212719 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:36.212733 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:36.212749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:36.212787 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:36.217238 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:36.218262 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:36.218270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:36.218273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:36.218549 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:36.219040 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:36.219044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:36.219046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:36.219494 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:36.219843 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:36.220251 (+ 408us) server_negotiation.cc:299] Negotiation successful
0811 12:36:36.220498 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:36:36.222054 26734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:36.322386 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 26469
I20260811 12:36:36.353214 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 26746
I20260811 12:36:36.369180 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:36.354049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.3.204.66:45303)
0811 12:36:36.354209 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:36.354214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:36.354232 (+ 18us) server_negotiation.cc:407] Connection header received
0811 12:36:36.354382 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:36.354385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:36.354451 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:36.354538 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:36.356222 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.356939 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.358829 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.359577 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.359679 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:36.363056 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:36.363071 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:36.363089 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:36.363127 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:36.366560 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:36.367246 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:36.367253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:36.367256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:36.367381 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:36.367673 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:36.367676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:36.367678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:36.368036 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:36.368331 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:36.368634 (+ 303us) server_negotiation.cc:299] Negotiation successful
0811 12:36:36.368824 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 12:36:36.369964 26975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:36.461933 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 26746
I20260811 12:36:36.497169 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 26992
I20260811 12:36:36.510730 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:36.497979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.3.204.67:37623)
0811 12:36:36.498202 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:36.498209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:36.498545 (+ 336us) server_negotiation.cc:407] Connection header received
0811 12:36:36.498728 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:36.498735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:36.498828 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:36.498918 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:36.500215 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.500975 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.502228 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:36.502599 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:36.502691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:36.505374 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:36.505392 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:36.505410 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:36.505453 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:36.508309 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:36.508991 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:36.508998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:36.509001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:36.509133 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:36.509437 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:36.509440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:36.509442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:36.509796 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:36.510028 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:36.510247 (+ 219us) server_negotiation.cc:299] Negotiation successful
0811 12:36:36.510411 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 12:36:36.511417 27213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:36.599884 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 26992
I20260811 12:36:36.620957 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 26339
I20260811 12:36:36.622195 26459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:36.733824 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 26339
I20260811 12:36:36.769232 3889 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 12:36:37.116828 3889 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 12:36:37.117048 3889 mini_ranger.cc:66] Stopping Ranger...
I20260811 12:36:37.462939 3889 mini_ranger.cc:68] Stopped Ranger
2026-08-11 12:36:37.463 UTC [24677] LOG: received smart shutdown request
2026-08-11 12:36:37.466 UTC [24677] LOG: background worker "logical replication launcher" (PID 24696) exited with exit code 1
2026-08-11 12:36:37.466 UTC [24687] LOG: shutting down
2026-08-11 12:36:37.624 UTC [24677] LOG: database system is shut down
2026-08-11T12:36:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53608 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:36:37 dist-test-slave-lgr5 krb5kdc[27342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:36:37 dist-test-slave-lgr5 krb5kdc[27342](info): set up 2 sockets
Aug 11 12:36:37 dist-test-slave-lgr5 krb5kdc[27342](info): commencing operation
krb5kdc: starting...
W20260811 12:36:39.197479 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.426s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260811 12:36:39.306463 3889 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.108s user 0.000s sys 0.034s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260811 12:36:39.410671 3889 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.103s user 0.004s sys 0.025s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260811 12:36:39.535926 3889 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.125s user 0.000s sys 0.035s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/test-user.keytab.
W20260811 12:36:39.641434 3889 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.105s user 0.003s sys 0.040s
Aug 11 12:36:39 dist-test-slave-lgr5 krb5kdc[27342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260811 12:36:39.789585 3889 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.148s user 0.006s sys 0.072s
2026-08-11T12:36:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:36:39Z Disabled control of system clock
W20260811 12:36:39.905593 3889 mini_chronyd.cc:189] Time spent starting chronyd: real 0.115s user 0.000s sys 0.058s
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:36:40.007897 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:43605
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:43605,127.3.204.125:43065,127.3.204.124:36893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:40.355273 27362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:40.355587 27362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:40.355721 27362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:40.363118 27362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:40.363236 27362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:40.363294 27362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:40.363382 27362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:40.363444 27362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:40.373519 27362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:43605,127.3.204.125:43065,127.3.204.124:36893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:43605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:40.374940 27362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:40.376291 27362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:40.388424 27371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:40.388470 27375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:40.388396 27373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:40.389446 27362 server_base.cc:1034] running on GCE node
I20260811 12:36:40.390273 27362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:40.391826 27362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:40.393060 27362 hybrid_clock.cc:648] HybridClock initialized: now 1786451800392988 us; error 68 us; skew 500 ppm
I20260811 12:36:40.393446 27362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:40 dist-test-slave-lgr5 krb5kdc[27342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:40.400112 27362 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:36:40.401958 27362 webserver.cc:460] Webserver started at http://127.3.204.126:37113/ using document root <none> and password file <none>
I20260811 12:36:40.402746 27362 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:40.402885 27362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:40.403215 27362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:36:40.406515 27362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f"
format_stamp: "Formatted at 2026-08-11 12:36:40 on dist-test-slave-lgr5"
server_key: "183c619e0e3a2d990770b305795aaea0"
server_key_iv: "43b9509e261af9bae1dfb491c5b4d8b4"
server_key_version: "encryptionkey@0"
I20260811 12:36:40.407377 27362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f"
format_stamp: "Formatted at 2026-08-11 12:36:40 on dist-test-slave-lgr5"
server_key: "183c619e0e3a2d990770b305795aaea0"
server_key_iv: "43b9509e261af9bae1dfb491c5b4d8b4"
server_key_version: "encryptionkey@0"
I20260811 12:36:40.412858 27362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:36:40.416432 27388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:40.417855 27362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 12:36:40.418030 27362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f"
format_stamp: "Formatted at 2026-08-11 12:36:40 on dist-test-slave-lgr5"
server_key: "183c619e0e3a2d990770b305795aaea0"
server_key_iv: "43b9509e261af9bae1dfb491c5b4d8b4"
server_key_version: "encryptionkey@0"
I20260811 12:36:40.418226 27362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:40.437880 27362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:40.442348 27362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:40.442636 27362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:40.460846 27362 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:43605
I20260811 12:36:40.460897 27480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:43605 every 8 connection(s)
I20260811 12:36:40.462292 27362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:36:40.465698 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 27362
I20260811 12:36:40.465937 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:36:40.466326 27482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:40.466782 3889 external_mini_cluster.cc:1428] Setting key 32164bb4241007b32d5a992f5370848a
I20260811 12:36:40.475636 27482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:36:40 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.204.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.125@KRBTEST.COM" created.
I20260811 12:36:40.498664 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.479684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43605 (local address 127.0.0.1:39434)
0811 12:36:40.480711 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.480758 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:40.482516 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:40.483406 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:40.483413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:40.484180 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:40.484546 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:40.484565 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:40.486105 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:40.486109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:40.487189 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:40.487201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:40.488133 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:40.488141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:40.488369 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:40.489123 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:40.489147 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:40.491373 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:40.494704 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:40.494716 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:40.494726 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:40.495171 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:40.495603 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:40.495607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:40.495609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:40.495766 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:40.496675 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:40.496682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:40.496881 (+ 199us) client_negotiation.cc:769] Sending connection context
0811 12:36:40.497101 (+ 220us) client_negotiation.cc:240] Negotiation successful
0811 12:36:40.497539 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":39,"thread_start_us":422,"threads_started":1}
I20260811 12:36:40.498721 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.479684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.3.204.126:43605)
0811 12:36:40.480649 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:40.480664 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:40.480870 (+ 206us) server_negotiation.cc:407] Connection header received
0811 12:36:40.482904 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:40.482910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:40.483173 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:40.483358 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:40.484741 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:40.485947 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:40.487322 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:40.487878 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:40.488063 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:40.491591 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:40.491600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:40.491613 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:40.491642 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:40.494551 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:40.495305 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:40.495314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:40.495317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:40.495504 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:40.495852 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:40.495855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:40.495857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:40.496505 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:40.496705 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:40.497310 (+ 605us) server_negotiation.cc:299] Negotiation successful
0811 12:36:40.497684 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":731,"spinlock_wait_cycles":49024,"thread_start_us":496,"threads_started":1}
I20260811 12:36:40.501609 27482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:36:40.502625 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.502234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42310)
0811 12:36:40.502403 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.502517 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 12:36:40.502858 27393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
I20260811 12:36:40.504395 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.504023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42322)
0811 12:36:40.504186 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.504261 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 12:36:40.504580 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:43065: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab.
Entry for principal kudu/127.3.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab.
I20260811 12:36:40.527637 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.125:43065
--webserver_interface=127.3.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab
--principal=kudu/127.3.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:43605,127.3.204.125:43065,127.3.204.124:36893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:36:40.554005 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } attempt: 1
I20260811 12:36:40.555313 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.554766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42336)
0811 12:36:40.555000 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.555151 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260811 12:36:40.556465 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.555979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42348)
0811 12:36:40.556151 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.556283 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 12:36:40.556669 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:43065: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
I20260811 12:36:40.624927 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } attempt: 2
I20260811 12:36:40.626293 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.625681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42358)
0811 12:36:40.625981 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.626106 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260811 12:36:40.627367 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.626952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42364)
0811 12:36:40.627111 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.627199 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 12:36:40.627559 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:43065: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
I20260811 12:36:40.718860 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } attempt: 3
I20260811 12:36:40.720409 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.719649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42380)
0811 12:36:40.719931 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.720127 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260811 12:36:40.721608 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.721204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42396)
0811 12:36:40.721355 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.721444 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 12:36:40.721807 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:43065: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
I20260811 12:36:40.865060 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } attempt: 4
I20260811 12:36:40.866364 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.865738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42410)
0811 12:36:40.866076 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.866186 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260811 12:36:40.867408 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.867011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42424)
0811 12:36:40.867154 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.867262 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 12:36:40.867606 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:43065: Network error: Client connection negotiation failed: client connection to 127.3.204.125:43065: connect: Connection refused (error 111)
W20260811 12:36:40.870779 27499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:40.871078 27499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:40.871194 27499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:40.878510 27499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:40.878603 27499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:40.878661 27499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:40.878716 27499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:40.878759 27499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:40.888638 27499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:43605,127.3.204.125:43065,127.3.204.124:36893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab
--principal=kudu/127.3.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.125:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:40.889926 27499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:40.891166 27499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:40.903092 27504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:40.903064 27511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:40.903021 27505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:40.904291 27499 server_base.cc:1034] running on GCE node
I20260811 12:36:40.905146 27499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:40.906646 27499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:40.907840 27499 hybrid_clock.cc:648] HybridClock initialized: now 1786451800907788 us; error 46 us; skew 500 ppm
I20260811 12:36:40.908242 27499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:40 dist-test-slave-lgr5 krb5kdc[27342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:40.911407 27499 init.cc:377] Logged in from keytab as kudu/127.3.204.125@KRBTEST.COM (short username kudu)
I20260811 12:36:40.913031 27499 webserver.cc:460] Webserver started at http://127.3.204.125:39093/ using document root <none> and password file <none>
I20260811 12:36:40.913735 27499 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:40.913852 27499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:40.914175 27499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:36:40.917316 27499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/data/instance:
uuid: "8dbde6555ba04eefb1346274f040af6f"
format_stamp: "Formatted at 2026-08-11 12:36:40 on dist-test-slave-lgr5"
server_key: "50d1088c2b88bf9959ab8e929943580d"
server_key_iv: "38814ad94ea4573ae7075d6d0a9f0499"
server_key_version: "encryptionkey@0"
I20260811 12:36:40.918123 27499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/wal/instance:
uuid: "8dbde6555ba04eefb1346274f040af6f"
format_stamp: "Formatted at 2026-08-11 12:36:40 on dist-test-slave-lgr5"
server_key: "50d1088c2b88bf9959ab8e929943580d"
server_key_iv: "38814ad94ea4573ae7075d6d0a9f0499"
server_key_version: "encryptionkey@0"
I20260811 12:36:40.923487 27499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:36:40.927098 27522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:40.928483 27499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:36:40.928645 27499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/wal
uuid: "8dbde6555ba04eefb1346274f040af6f"
format_stamp: "Formatted at 2026-08-11 12:36:40 on dist-test-slave-lgr5"
server_key: "50d1088c2b88bf9959ab8e929943580d"
server_key_iv: "38814ad94ea4573ae7075d6d0a9f0499"
server_key_version: "encryptionkey@0"
I20260811 12:36:40.928836 27499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:40.941241 27499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:40.945389 27499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:40.945659 27499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:40.963534 27499 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.125:43065
I20260811 12:36:40.963565 27607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.125:43065 every 8 connection(s)
I20260811 12:36:40.964929 27499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
I20260811 12:36:40.968700 27608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:40.975317 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 27499
I20260811 12:36:40.975910 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-1/wal/instance
I20260811 12:36:40.976531 3889 external_mini_cluster.cc:1428] Setting key 7afb22a601a295b37381a4b8b3697227
I20260811 12:36:40.976140 27608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:36:40 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:41.001555 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.979852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.3.204.126:43605)
0811 12:36:40.980094 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:40.980101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:40.981481 (+ 1380us) server_negotiation.cc:407] Connection header received
0811 12:36:40.984072 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:40.984115 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:40.984213 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:40.984321 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:40.986437 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:40.987341 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:40.989298 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:40.989903 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:40.990035 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:40.994049 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:40.994056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:40.994060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:40.994108 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:40.997830 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:40.998621 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:40.998626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:40.998630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:40.998779 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:40.999292 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:40.999298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:40.999302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:40.999715 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:40.999835 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.000982 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.001220 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260811 12:36:41.003123 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:40.980401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43605 (local address 127.0.0.1:39444)
0811 12:36:40.981286 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:40.981334 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:40.983738 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:40.984673 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:40.984684 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:40.985409 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:40.986193 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:40.986218 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:40.987555 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:40.987560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:40.989135 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:40.989151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:40.990112 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:40.990117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:40.990318 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:40.990974 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:40.990997 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:40.993772 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:40.997998 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:40.998008 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:40.998027 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:40.998484 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:40.998893 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:40.998899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:40.998903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:40.999149 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:40.999951 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:40.999960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:41.000553 (+ 593us) client_negotiation.cc:769] Sending connection context
0811 12:36:41.000870 (+ 317us) client_negotiation.cc:240] Negotiation successful
0811 12:36:41.001425 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":341,"threads_started":1}
I20260811 12:36:41.005654 27608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.204.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.124@KRBTEST.COM" created.
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:36:41.025290 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.006562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42440)
0811 12:36:41.006863 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.006884 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:41.007021 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:41.010234 (+ 3213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:41.010238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:41.010717 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:41.010871 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.010882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.012783 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.012789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.013974 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.013991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.015393 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.015399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.015612 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.016243 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:41.016260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:41.018769 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:41.022724 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.022728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.022731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.023144 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.023546 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.023550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.023553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.023704 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.024333 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:41.024342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:41.024478 (+ 136us) client_negotiation.cc:769] Sending connection context
0811 12:36:41.024596 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 12:36:41.024881 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260811 12:36:41.025368 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.006673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.3.204.125:43065)
0811 12:36:41.009783 (+ 3110us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.009789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.009817 (+ 28us) server_negotiation.cc:407] Connection header received
0811 12:36:41.009869 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.009876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.010078 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.010220 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.011066 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.012632 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.014126 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.015037 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.015172 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.019004 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.019018 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.019036 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.019076 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.022565 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.023297 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.023305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.023308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.023443 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.023789 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.023794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.023797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.024209 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.024390 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.024785 (+ 395us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.024997 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2940,"thread_start_us":2896,"threads_started":1}
I20260811 12:36:41.028426 27608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:36:41.029824 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.029238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46274)
0811 12:36:41.029506 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.029634 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260811 12:36:41.030104 27525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
I20260811 12:36:41.031739 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.031427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46282)
0811 12:36:41.031561 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.031633 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260811 12:36:41.032058 27608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab.
Entry for principal kudu/127.3.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab.
I20260811 12:36:41.049714 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.124:36893
--webserver_interface=127.3.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab
--principal=kudu/127.3.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:43605,127.3.204.125:43065,127.3.204.124:36893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:36:41.081437 27608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 1
I20260811 12:36:41.082664 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.082194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46288)
0811 12:36:41.082389 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.082505 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 12:36:41.083868 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.083406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46290)
0811 12:36:41.083625 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.083720 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260811 12:36:41.084125 27608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
I20260811 12:36:41.152366 27608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 2
I20260811 12:36:41.153527 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.153044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46306)
0811 12:36:41.153260 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.153358 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260811 12:36:41.154596 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.154277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46312)
0811 12:36:41.154400 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.154476 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 12:36:41.154821 27608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
I20260811 12:36:41.166900 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } attempt: 5
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:36:41.182020 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.167733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42450)
0811 12:36:41.167965 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.167988 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:41.168169 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:41.168637 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:41.168642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:41.169041 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:41.169236 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.169246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.170639 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.170644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.171638 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.171654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.172700 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.172705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.172877 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.173471 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:41.173491 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:41.175479 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:41.179292 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.179297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.179301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.179722 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.180191 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.180196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.180198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.180358 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.181109 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:41.181115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:41.181330 (+ 215us) client_negotiation.cc:769] Sending connection context
0811 12:36:41.181482 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 12:36:41.181745 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260811 12:36:41.182163 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.167789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.3.204.125:43065)
0811 12:36:41.167972 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.167978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.168126 (+ 148us) server_negotiation.cc:407] Connection header received
0811 12:36:41.168305 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.168312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.168434 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.168569 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.169404 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.170478 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.171793 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.172526 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.172670 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.175692 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.175699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.175703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.175745 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.179119 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.179850 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.179856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.179859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.180032 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.180478 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.180483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.180486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.180872 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.181029 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.181608 (+ 579us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.181847 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 12:36:41.183418 27482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:36:41.184885 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.184412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46326)
0811 12:36:41.184616 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.184702 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 12:36:41.185937 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.185584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46332)
0811 12:36:41.185749 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.185831 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 12:36:41.186143 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
I20260811 12:36:41.237396 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 1
I20260811 12:36:41.238659 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.238141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46334)
0811 12:36:41.238373 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.238488 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260811 12:36:41.239732 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.239376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46346)
0811 12:36:41.239543 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.239618 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 12:36:41.239952 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
I20260811 12:36:41.246037 27608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 3
I20260811 12:36:41.247082 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.246623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46362)
0811 12:36:41.246833 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.246933 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 12:36:41.248073 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.247673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46366)
0811 12:36:41.247886 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.247961 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260811 12:36:41.248281 27608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
I20260811 12:36:41.308212 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 2
I20260811 12:36:41.309351 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.308852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46368)
0811 12:36:41.309094 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.309185 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260811 12:36:41.310354 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.310045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46372)
0811 12:36:41.310187 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.310251 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 12:36:41.310575 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
I20260811 12:36:41.391522 27608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 4
I20260811 12:36:41.392791 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.392275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46382)
0811 12:36:41.392532 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.392625 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 12:36:41.393837 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.393458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46392)
0811 12:36:41.393609 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.393699 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 12:36:41.394134 27608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
W20260811 12:36:41.401635 27621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:41.401930 27621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:41.402069 27621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:41.409716 27621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:41.409845 27621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:41.409924 27621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:41.410006 27621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:41.410061 27621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:41.416771 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 3
I20260811 12:36:41.417966 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.417473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46408)
0811 12:36:41.417706 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.417799 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 12:36:41.418947 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.418600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46424)
0811 12:36:41.418763 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.418842 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260811 12:36:41.419145 27482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:36893: Network error: Client connection negotiation failed: client connection to 127.3.204.124:36893: connect: Connection refused (error 111)
I20260811 12:36:41.420907 27621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:43605,127.3.204.125:43065,127.3.204.124:36893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab
--principal=kudu/127.3.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.124:36893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:41.422358 27621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:41.423735 27621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:41.436774 27633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:41.437230 27631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:41.436724 27635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:41.437868 27621 server_base.cc:1034] running on GCE node
I20260811 12:36:41.438687 27621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:41.440166 27621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:41.441383 27621 hybrid_clock.cc:648] HybridClock initialized: now 1786451801441335 us; error 36 us; skew 500 ppm
I20260811 12:36:41.441766 27621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:41.444942 27621 init.cc:377] Logged in from keytab as kudu/127.3.204.124@KRBTEST.COM (short username kudu)
I20260811 12:36:41.446483 27621 webserver.cc:460] Webserver started at http://127.3.204.124:45983/ using document root <none> and password file <none>
I20260811 12:36:41.447185 27621 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:41.447295 27621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:41.447618 27621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:36:41.450706 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/data/instance:
uuid: "50abe599046e446f985960b1f537899d"
format_stamp: "Formatted at 2026-08-11 12:36:41 on dist-test-slave-lgr5"
server_key: "20b0a4ca37df3c4feabd658a17ea48cc"
server_key_iv: "38e2eb20ac59fc8c5ab0f97a7cc4d02c"
server_key_version: "encryptionkey@0"
I20260811 12:36:41.451520 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/wal/instance:
uuid: "50abe599046e446f985960b1f537899d"
format_stamp: "Formatted at 2026-08-11 12:36:41 on dist-test-slave-lgr5"
server_key: "20b0a4ca37df3c4feabd658a17ea48cc"
server_key_iv: "38e2eb20ac59fc8c5ab0f97a7cc4d02c"
server_key_version: "encryptionkey@0"
I20260811 12:36:41.456722 27621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260811 12:36:41.460229 27647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:41.461625 27621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:36:41.461781 27621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/wal
uuid: "50abe599046e446f985960b1f537899d"
format_stamp: "Formatted at 2026-08-11 12:36:41 on dist-test-slave-lgr5"
server_key: "20b0a4ca37df3c4feabd658a17ea48cc"
server_key_iv: "38e2eb20ac59fc8c5ab0f97a7cc4d02c"
server_key_version: "encryptionkey@0"
I20260811 12:36:41.461967 27621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:41.489223 27621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:41.493685 27621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:41.493958 27621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:41.515858 27621 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.124:36893
I20260811 12:36:41.515908 27735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.124:36893 every 8 connection(s)
I20260811 12:36:41.517292 27621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
I20260811 12:36:41.518405 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 27621
I20260811 12:36:41.518584 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786451585252647-3889-0/minicluster-data/master-2/wal/instance
I20260811 12:36:41.519090 3889 external_mini_cluster.cc:1428] Setting key 0a9a8ee01df51665c0974fa03dc062e6
I20260811 12:36:41.521855 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:41.530982 27739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:41.544986 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.521036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.3.204.124:36893)
0811 12:36:41.521926 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.521942 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.521995 (+ 53us) server_negotiation.cc:407] Connection header received
0811 12:36:41.524334 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.524365 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.524870 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.525342 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.526799 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.528893 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.530332 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.531391 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.531607 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.537023 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.537031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.537042 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.537080 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.540427 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.541377 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.541386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.541395 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.541633 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.542128 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.542134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.542138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.542815 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.543092 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.543545 (+ 453us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.543910 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":342,"threads_started":1}
I20260811 12:36:41.549707 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.534061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43605 (local address 127.0.0.1:39448)
0811 12:36:41.535139 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.535191 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:41.535382 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:41.535793 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:41.535802 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:41.536660 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:41.537047 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.537060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.538450 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.538454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.539689 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.539703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.540730 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.540736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.540894 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.541532 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:41.541553 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:41.543858 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:41.547022 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.547034 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.547041 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.547584 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.547929 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.547935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.547939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.548138 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.548786 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:41.548793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:41.548990 (+ 197us) client_negotiation.cc:769] Sending connection context
0811 12:36:41.549200 (+ 210us) client_negotiation.cc:240] Negotiation successful
0811 12:36:41.549432 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":683,"threads_started":1}
I20260811 12:36:41.550005 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.534059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.3.204.126:43605)
0811 12:36:41.534613 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.534618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.535297 (+ 679us) server_negotiation.cc:407] Connection header received
0811 12:36:41.535517 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.535521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.535591 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.535693 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.537210 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.538270 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.539855 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.540565 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.540674 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.544029 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.544033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.544037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.544070 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.546898 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.547696 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.547700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.547702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.547824 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.548263 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.548268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.548271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.548642 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.548762 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.549381 (+ 619us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.549602 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":311,"threads_started":1}
I20260811 12:36:41.551354 27739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:36:41.563369 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.548070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.3.204.126:43605)
0811 12:36:41.548692 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.548698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.548721 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:36:41.548768 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.548772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.548840 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.548956 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.550458 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.551543 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.553260 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.553668 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.553785 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.557985 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.557990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.557993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.558024 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.560692 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.561367 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.561372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.561374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.561497 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.561913 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.561917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.561919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.562231 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.562537 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.562738 (+ 201us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.562964 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":296,"threads_started":1}
I20260811 12:36:41.564189 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.552106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:43065 (local address 127.0.0.1:42452)
0811 12:36:41.552268 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.552287 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:41.552385 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:41.552670 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:41.552675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:41.553109 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:41.553301 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.553312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.554232 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.554239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.555131 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.555143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.555842 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.555848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.556049 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.556595 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:41.556618 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:41.558749 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:41.561660 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.561665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.561669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.562088 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.562519 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.562524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.562527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.562711 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.563288 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:41.563291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:41.563414 (+ 123us) client_negotiation.cc:769] Sending connection context
0811 12:36:41.563551 (+ 137us) client_negotiation.cc:240] Negotiation successful
0811 12:36:41.563820 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 12:36:41.564816 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.552145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.3.204.125:43065)
0811 12:36:41.552326 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.552330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.552346 (+ 16us) server_negotiation.cc:407] Connection header received
0811 12:36:41.552500 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.552504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.552562 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.552627 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.553457 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.554099 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.555273 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.555686 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.555795 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.558960 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.558967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.558971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.559007 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.561548 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.562210 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.562216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.562219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.562382 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.562824 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.562829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.562832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.563177 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.563334 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.564223 (+ 889us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.564567 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 12:36:41.565743 27739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:36:41.578281 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.566697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.3.204.124:36893)
0811 12:36:41.566842 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.566847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.566861 (+ 14us) server_negotiation.cc:407] Connection header received
0811 12:36:41.566896 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.566899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.566945 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.567021 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.567649 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.568392 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.569474 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.569888 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.569994 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.572751 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.572756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.572759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.572787 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.575637 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.576285 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.576290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.576293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.576441 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.576781 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.576786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.576790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.577149 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.577252 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.577700 (+ 448us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.577936 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 12:36:41.578593 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.566470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46442)
0811 12:36:41.566622 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.566639 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:41.566729 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:41.567053 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:41.567057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:41.567387 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:41.567515 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.567523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.568515 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.568519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.569340 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.569351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.570040 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.570046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.570227 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.570680 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:41.570696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:41.572579 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:41.575773 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.575777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.575780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.576175 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.576557 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.576561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.576563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.576685 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.577267 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:41.577272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:41.577438 (+ 166us) client_negotiation.cc:769] Sending connection context
0811 12:36:41.577626 (+ 188us) client_negotiation.cc:240] Negotiation successful
0811 12:36:41.578125 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":155}
I20260811 12:36:41.579933 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.565358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.3.204.125:43065)
0811 12:36:41.565525 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.565530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.565546 (+ 16us) server_negotiation.cc:407] Connection header received
0811 12:36:41.565591 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.565595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.565658 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.565748 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.567177 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.567890 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.569170 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.569606 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.569714 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.574020 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.574027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.574031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.574070 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.576968 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.577730 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.577735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.577738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.577898 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.578319 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.578324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.578327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.578671 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.578943 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.579454 (+ 511us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.579693 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 12:36:41.584234 27739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d: Bootstrap starting.
I20260811 12:36:41.584713 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:36:41.587486 27739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:41.588644 27739 log.cc:824] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:41.597316 27482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 4
I20260811 12:36:41.610698 27739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d: No bootstrap required, opened a new log
I20260811 12:36:41.613125 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.589312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.3.204.125:43065)
0811 12:36:41.589675 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.589683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.593748 (+ 4065us) server_negotiation.cc:407] Connection header received
0811 12:36:41.594261 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.594268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.594360 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.594432 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.597236 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.598423 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.600750 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.601465 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.601576 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.605780 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.605788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.605794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.605858 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.609339 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.610353 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.610359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.610362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.610528 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.611165 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.611171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.611174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.611580 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.611804 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.612510 (+ 706us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.612759 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260811 12:36:41.618103 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.589282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.3.204.126:43605)
0811 12:36:41.589847 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.589854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.595865 (+ 6011us) server_negotiation.cc:407] Connection header received
0811 12:36:41.596017 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.596023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.596178 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.596273 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.598391 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.599439 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.601672 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.602342 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.602432 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.607544 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.607551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.607556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.607597 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.611191 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.613072 (+ 1881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.613078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.613081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.613240 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.616231 (+ 2991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.616238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.616241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.616616 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.616882 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.617472 (+ 590us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.617711 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260811 12:36:41.619163 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.589462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.3.204.124:36893)
0811 12:36:41.590575 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.590582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.591915 (+ 1333us) server_negotiation.cc:407] Connection header received
0811 12:36:41.592475 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.592482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.592571 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.592646 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.596061 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.597173 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.607746 (+ 10573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.608512 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.608616 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.612001 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.612008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.612013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.612051 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.615699 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.616517 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.616522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.616525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.616713 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.617149 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.617154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.617157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.617515 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.617734 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.618528 (+ 794us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.618768 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942}
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
I20260811 12:36:41.622073 27739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.622435 27739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:41.622583 27739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50abe599046e446f985960b1f537899d, State: Initialized, Role: FOLLOWER
I20260811 12:36:41.623320 27739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.624979 27765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.625444 27765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:36:41.626096 27739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:41.628902 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.597968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46466)
0811 12:36:41.598323 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.598342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:41.598423 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:41.607261 (+ 8838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:41.607267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:41.607816 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:41.608051 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.608065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.616654 (+ 8589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.616660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.618194 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.618211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.619265 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.619273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.619444 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.620052 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:41.620074 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:41.622435 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:41.626128 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.626134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.626138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.626542 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.626910 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.626915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.626920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.627083 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.627844 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:41.627849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:41.628041 (+ 192us) client_negotiation.cc:769] Sending connection context
0811 12:36:41.628227 (+ 186us) client_negotiation.cc:240] Negotiation successful
0811 12:36:41.628519 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260811 12:36:41.629014 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.598747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.3.204.124:36893)
0811 12:36:41.606878 (+ 8131us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.606884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.606908 (+ 24us) server_negotiation.cc:407] Connection header received
0811 12:36:41.606984 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.606989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.607064 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.607186 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.612054 (+ 4868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.616489 (+ 4435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.618374 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.619084 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.621392 (+ 2308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.622669 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.622676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.622681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.622715 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.626003 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.626652 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.626661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.626664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.626796 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.627221 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.627227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.627230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.627687 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.627850 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.628369 (+ 519us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.628565 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7966,"thread_start_us":7776,"threads_started":1}
I20260811 12:36:41.630438 27765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:36:41.630784 27765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.632452 27765 leader_election.cc:288] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fd7a437a6ae4e05b0eb6d765de5fe1f (127.3.204.126:43605), 8dbde6555ba04eefb1346274f040af6f (127.3.204.125:43065)
I20260811 12:36:41.635622 27482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: Bootstrap starting.
I20260811 12:36:41.639839 27482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:41.641152 27482 log.cc:824] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: Log is configured to *not* fsync() on all Append() calls
W20260811 12:36:41.641798 27782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 12:36:41.641918 27782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 12:36:41.644977 27482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: No bootstrap required, opened a new log
I20260811 12:36:41.638370 27621 master_runner.cc:186] Error getting master registration for 127.3.204.126:43605: OK, instance_id { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" instance_seqno: 1786451800455642 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 12:36:41.649152 27621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 12:36:41.656390 27482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.656832 27482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:41.656957 27482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd7a437a6ae4e05b0eb6d765de5fe1f, State: Initialized, Role: FOLLOWER
I20260811 12:36:41.657676 27482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.658690 27791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.659155 27791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:36:41.660207 27482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:41.661137 27437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50abe599046e446f985960b1f537899d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" is_pre_election: true
I20260811 12:36:41.661782 27437 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50abe599046e446f985960b1f537899d in term 0.
I20260811 12:36:41.662588 27654 leader_election.cc:302] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50abe599046e446f985960b1f537899d, 5fd7a437a6ae4e05b0eb6d765de5fe1f; no voters:
I20260811 12:36:41.663022 27765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:36:41.663183 27765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:36:41.663287 27765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:41.666836 27765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.667591 27765 leader_election.cc:288] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [CANDIDATE]: Term 1 election: Requested vote from peers 5fd7a437a6ae4e05b0eb6d765de5fe1f (127.3.204.126:43605), 8dbde6555ba04eefb1346274f040af6f (127.3.204.125:43065)
I20260811 12:36:41.668515 27437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50abe599046e446f985960b1f537899d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f"
I20260811 12:36:41.668748 27437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:41.672065 27437 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50abe599046e446f985960b1f537899d in term 1.
I20260811 12:36:41.673300 27654 leader_election.cc:302] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50abe599046e446f985960b1f537899d, 5fd7a437a6ae4e05b0eb6d765de5fe1f; no voters:
I20260811 12:36:41.673719 27765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:36:41.674861 27765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [term 1 LEADER]: Becoming Leader. State: Replica: 50abe599046e446f985960b1f537899d, State: Running, Role: LEADER
I20260811 12:36:41.676014 27765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.679924 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50abe599046e446f985960b1f537899d. Latest consensus state: current_term: 1 leader_uuid: "50abe599046e446f985960b1f537899d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.680431 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:41.692934 27817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:36:41.693382 27608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } attempt: 5
W20260811 12:36:41.696609 27397 tablet.cc:2369] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 12:36:41 dist-test-slave-lgr5 krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
I20260811 12:36:41.704368 27817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:36:41.718308 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.694061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:36893 (local address 127.0.0.1:46476)
0811 12:36:41.694490 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:41.694512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:41.694627 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:41.695418 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:41.695423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:41.695969 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:41.696208 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.696221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.698782 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.698788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.700211 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:41.700226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.701187 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.701193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:41.701461 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.702104 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:41.702128 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:41.704877 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:41.713130 (+ 8253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.713138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.713143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.713791 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.714716 (+ 925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:41.714750 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:41.714755 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:41.715111 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:41.716446 (+ 1335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:41.716478 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:41.716759 (+ 281us) client_negotiation.cc:769] Sending connection context
0811 12:36:41.717021 (+ 262us) client_negotiation.cc:240] Negotiation successful
0811 12:36:41.717608 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260811 12:36:41.708827 27362 master_runner.cc:186] Error getting master registration for 127.3.204.125:43065: OK, instance_id { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" instance_seqno: 1786451800958370 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 12:36:41.719463 27362 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 12:36:41.719991 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:41.694558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.3.204.124:36893)
0811 12:36:41.694911 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:41.694917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:41.694937 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:36:41.695009 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:41.695014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:41.695111 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:41.695218 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:41.696399 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.698651 (+ 2252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.700352 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:41.701031 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:41.701157 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:41.705247 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:41.705257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:41.705263 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:41.705347 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:41.712762 (+ 7415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.714144 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.714151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.714155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.714490 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:41.715391 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:41.715397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:41.715401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:41.716250 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:41.718908 (+ 2658us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:41.719204 (+ 296us) server_negotiation.cc:299] Negotiation successful
0811 12:36:41.719669 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
W20260811 12:36:41.722522 27821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 12:36:41.722640 27821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 12:36:41.724061 27437 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [term 1 FOLLOWER]: Refusing update from remote peer 50abe599046e446f985960b1f537899d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:36:41.725028 27809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:36:41.726356 27608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f: Bootstrap starting.
I20260811 12:36:41.730404 27608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:41.731837 27608 log.cc:824] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:41.735273 27791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50abe599046e446f985960b1f537899d. Latest consensus state: current_term: 1 leader_uuid: "50abe599046e446f985960b1f537899d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.735635 27791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:36:41.741649 27608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f: No bootstrap required, opened a new log
I20260811 12:36:41.754480 27608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.755088 27608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:41.755272 27608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dbde6555ba04eefb1346274f040af6f, State: Initialized, Role: FOLLOWER
I20260811 12:36:41.755792 27765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50abe599046e446f985960b1f537899d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.756184 27765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:41.756292 27608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } }
I20260811 12:36:41.759071 27817 catalog_manager.cc:1347] Generated new cluster ID: 201bc14f58a744dba1385f2a73c56266
I20260811 12:36:41.759183 27817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:36:41.763376 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.763797 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:36:41.764971 27791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50abe599046e446f985960b1f537899d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.765220 27791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:36:41.765928 27608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:41.766352 27565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:41.767009 27567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50abe599046e446f985960b1f537899d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dbde6555ba04eefb1346274f040af6f" is_pre_election: true
I20260811 12:36:41.767992 27566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50abe599046e446f985960b1f537899d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dbde6555ba04eefb1346274f040af6f"
I20260811 12:36:41.770054 27565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [term 1 FOLLOWER]: Refusing update from remote peer 50abe599046e446f985960b1f537899d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:36:41.770797 27809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:36:41.808184 27817 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:36:41.810678 27817 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:36:41.821478 27851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50abe599046e446f985960b1f537899d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.821995 27851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:36:41.821532 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50abe599046e446f985960b1f537899d. Latest consensus state: current_term: 1 leader_uuid: "50abe599046e446f985960b1f537899d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.822664 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:36:41.822887 27765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50abe599046e446f985960b1f537899d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd7a437a6ae4e05b0eb6d765de5fe1f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43605 } } peers { permanent_uuid: "8dbde6555ba04eefb1346274f040af6f" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 43065 } } peers { permanent_uuid: "50abe599046e446f985960b1f537899d" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 36893 } } }
I20260811 12:36:41.823216 27765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:41.837227 27817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50abe599046e446f985960b1f537899d: Generated new TSK 0
I20260811 12:36:41.842106 27817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:36:41.843967 27862 catalog_manager.cc:1259] Loaded cluster ID: 201bc14f58a744dba1385f2a73c56266
I20260811 12:36:41.844174 27862 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f: loading cluster ID for follower catalog manager: success
I20260811 12:36:41.847916 27862 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f: acquiring CA information for follower catalog manager: success
I20260811 12:36:41.850391 27862 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8dbde6555ba04eefb1346274f040af6f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 12:36:42.724802 27821 catalog_manager.cc:1259] Loaded cluster ID: 201bc14f58a744dba1385f2a73c56266
I20260811 12:36:42.724910 27821 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: loading cluster ID for follower catalog manager: success
I20260811 12:36:42.727418 27821 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: acquiring CA information for follower catalog manager: success
I20260811 12:36:42.729369 27821 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5fd7a437a6ae4e05b0eb6d765de5fe1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 12:36:42.971561 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 27362
I20260811 12:36:42.972847 27470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:43.082218 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 27362
I20260811 12:36:43.112380 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 27499
I20260811 12:36:43.113512 27603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:43.220882 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 27499
I20260811 12:36:43.245726 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 27621
I20260811 12:36:43.246959 27724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:43.354624 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 27621
2026-08-11T12:36:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5631 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:36:43 dist-test-slave-lgr5 krb5kdc[27882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:36:43 dist-test-slave-lgr5 krb5kdc[27882](info): set up 2 sockets
Aug 11 12:36:43 dist-test-slave-lgr5 krb5kdc[27882](info): commencing operation
krb5kdc: starting...
W20260811 12:36:44.627663 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:36:44 dist-test-slave-lgr5 krb5kdc[27882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:36:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:36:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:36:44.915863 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:38213
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:45.260668 27902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:45.261016 27902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:45.261152 27902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:45.268582 27902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:45.268720 27902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:45.268797 27902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:45.268873 27902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:45.268934 27902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:45.279024 27902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:45.280483 27902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:45.281839 27902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:45.293538 27916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:45.293663 27911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:45.293672 27909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:45.294932 27902 server_base.cc:1034] running on GCE node
I20260811 12:36:45.295827 27902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:45.297362 27902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:45.298616 27902 hybrid_clock.cc:648] HybridClock initialized: now 1786451805298556 us; error 47 us; skew 500 ppm
I20260811 12:36:45.298996 27902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:45 dist-test-slave-lgr5 krb5kdc[27882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451805, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:45.302152 27902 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:36:45.303812 27902 webserver.cc:460] Webserver started at http://127.3.204.126:45165/ using document root <none> and password file <none>
I20260811 12:36:45.304587 27902 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:45.304709 27902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:45.305022 27902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:36:45.308145 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "7ef49bdbabc84ba09fbb99957078637b"
format_stamp: "Formatted at 2026-08-11 12:36:45 on dist-test-slave-lgr5"
server_key: "59ddc6915816ec2483e8af35c5d76d2d"
server_key_iv: "5a9bcd80d2cd2ef9cabcc0442b9cd02e"
server_key_version: "encryptionkey@0"
I20260811 12:36:45.308990 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "7ef49bdbabc84ba09fbb99957078637b"
format_stamp: "Formatted at 2026-08-11 12:36:45 on dist-test-slave-lgr5"
server_key: "59ddc6915816ec2483e8af35c5d76d2d"
server_key_iv: "5a9bcd80d2cd2ef9cabcc0442b9cd02e"
server_key_version: "encryptionkey@0"
I20260811 12:36:45.314529 27902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:36:45.318141 27932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:45.319520 27902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:36:45.319700 27902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "7ef49bdbabc84ba09fbb99957078637b"
format_stamp: "Formatted at 2026-08-11 12:36:45 on dist-test-slave-lgr5"
server_key: "59ddc6915816ec2483e8af35c5d76d2d"
server_key_iv: "5a9bcd80d2cd2ef9cabcc0442b9cd02e"
server_key_version: "encryptionkey@0"
I20260811 12:36:45.319890 27902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:45.337934 27902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:45.342017 27902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:45.342291 27902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:45.360142 27902 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:38213
I20260811 12:36:45.360199 28022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:38213 every 8 connection(s)
I20260811 12:36:45.361564 27902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:36:45.364575 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 27902
I20260811 12:36:45.364812 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:36:45.365695 3889 external_mini_cluster.cc:1428] Setting key 73f7ecbb723cc60ea9c2851feffd4707
I20260811 12:36:45.365887 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:45.378986 28023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b: Bootstrap starting.
Aug 11 12:36:45 dist-test-slave-lgr5 krb5kdc[27882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:45.382359 28023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:45.383783 28023 log.cc:824] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:45.387231 28023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b: No bootstrap required, opened a new log
I20260811 12:36:45.390257 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:45.367599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.3.204.126:38213)
0811 12:36:45.368386 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:45.368397 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:45.368439 (+ 42us) server_negotiation.cc:407] Connection header received
0811 12:36:45.371006 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:45.371037 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:45.371620 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:45.372048 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:45.373665 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:45.374932 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:45.376388 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:45.377110 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:45.377297 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:45.381708 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:45.381717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:45.381732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:45.381781 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:45.385390 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:45.386183 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:45.386191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:45.386200 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:45.386428 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:45.386858 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:45.386863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:45.386866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:45.387603 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:45.387931 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:45.388333 (+ 402us) server_negotiation.cc:299] Negotiation successful
0811 12:36:45.388783 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":352,"threads_started":1}
I20260811 12:36:45.394410 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:36:45.396849 28023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef49bdbabc84ba09fbb99957078637b" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38213 } }
I20260811 12:36:45.397326 28023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:45.397446 28023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef49bdbabc84ba09fbb99957078637b, State: Initialized, Role: FOLLOWER
I20260811 12:36:45.398022 28023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef49bdbabc84ba09fbb99957078637b" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38213 } }
I20260811 12:36:45.398238 28023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:36:45.398379 28023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:36:45.398550 28023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:45.400856 28023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef49bdbabc84ba09fbb99957078637b" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38213 } }
I20260811 12:36:45.401364 28023 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ef49bdbabc84ba09fbb99957078637b; no voters:
I20260811 12:36:45.401976 28023 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:36:45.402134 28034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:36:45.403000 28034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [term 1 LEADER]: Becoming Leader. State: Replica: 7ef49bdbabc84ba09fbb99957078637b, State: Running, Role: LEADER
I20260811 12:36:45.403565 28034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef49bdbabc84ba09fbb99957078637b" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38213 } }
I20260811 12:36:45.404208 28023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:45.406626 28036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef49bdbabc84ba09fbb99957078637b. Latest consensus state: current_term: 1 leader_uuid: "7ef49bdbabc84ba09fbb99957078637b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef49bdbabc84ba09fbb99957078637b" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38213 } } }
I20260811 12:36:45.406649 28035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ef49bdbabc84ba09fbb99957078637b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef49bdbabc84ba09fbb99957078637b" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 38213 } } }
I20260811 12:36:45.407078 28036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:45.407081 28035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:45.408907 28040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:36:45.414559 28040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:36:45.424584 28040 catalog_manager.cc:1347] Generated new cluster ID: ea0ce3c5e4414298ac76cf137c3e8bde
I20260811 12:36:45.424707 28040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:36:45.441887 28040 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:36:45.443554 28040 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:36:45.450860 28040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef49bdbabc84ba09fbb99957078637b: Generated new TSK 0
I20260811 12:36:45.451704 28040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:36:45.507545 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 27902
I20260811 12:36:45.508754 28014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:45.605480 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 27902
2026-08-11T12:36:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2248 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:36:45 dist-test-slave-lgr5 krb5kdc[28071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:36:45 dist-test-slave-lgr5 krb5kdc[28071](info): set up 2 sockets
Aug 11 12:36:45 dist-test-slave-lgr5 krb5kdc[28071](info): commencing operation
krb5kdc: starting...
W20260811 12:36:46.883438 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:36:47 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:36:47.176586 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46081
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:46081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:47.521046 28096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:47.521368 28096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:47.521467 28096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:47.528817 28096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:47.528951 28096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:47.528998 28096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:47.529058 28096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:47.529101 28096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:47.539296 28096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:46081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:47.540640 28096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:47.541893 28096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:47.553669 28102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:47.553812 28101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:47.553716 28106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:47.554656 28096 server_base.cc:1034] running on GCE node
I20260811 12:36:47.555413 28096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:47.556864 28096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:47.558076 28096 hybrid_clock.cc:648] HybridClock initialized: now 1786451807558024 us; error 36 us; skew 500 ppm
I20260811 12:36:47.558430 28096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:47 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:47.561571 28096 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:36:47.563095 28096 webserver.cc:460] Webserver started at http://127.3.204.126:46727/ using document root <none> and password file <none>
I20260811 12:36:47.563828 28096 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:47.563936 28096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:47.564277 28096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:36:47.567334 28096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8"
format_stamp: "Formatted at 2026-08-11 12:36:47 on dist-test-slave-lgr5"
server_key: "12538c978bf6a3fc857344254f6fce75"
server_key_iv: "84a3f6302c8210fcbef72b1369a1215a"
server_key_version: "encryptionkey@0"
I20260811 12:36:47.568159 28096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8"
format_stamp: "Formatted at 2026-08-11 12:36:47 on dist-test-slave-lgr5"
server_key: "12538c978bf6a3fc857344254f6fce75"
server_key_iv: "84a3f6302c8210fcbef72b1369a1215a"
server_key_version: "encryptionkey@0"
I20260811 12:36:47.573700 28096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 12:36:47.577364 28119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:47.578816 28096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:36:47.579005 28096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8"
format_stamp: "Formatted at 2026-08-11 12:36:47 on dist-test-slave-lgr5"
server_key: "12538c978bf6a3fc857344254f6fce75"
server_key_iv: "84a3f6302c8210fcbef72b1369a1215a"
server_key_version: "encryptionkey@0"
I20260811 12:36:47.579198 28096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:47.602645 28096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:47.606976 28096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:47.607250 28096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:47.624464 28096 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:46081
I20260811 12:36:47.624502 28224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:46081 every 8 connection(s)
I20260811 12:36:47.625869 28096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:36:47.629755 28225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:47.635290 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 28096
I20260811 12:36:47.635507 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:36:47.636181 3889 external_mini_cluster.cc:1428] Setting key 3879a6bda1dc89d6af596e0f6545e45f
I20260811 12:36:47.640954 28225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Bootstrap starting.
I20260811 12:36:47.644737 28225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:47.646057 28225 log.cc:824] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:47.649674 28225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: No bootstrap required, opened a new log
Aug 11 12:36:47 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:47.660071 28225 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:47.660612 28225 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:47.660727 28225 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad60ebf94bc94228a8b1a1c4fd255ea8, State: Initialized, Role: FOLLOWER
I20260811 12:36:47.661445 28225 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:47.661652 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:47.638258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.3.204.126:46081)
0811 12:36:47.639093 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:47.639105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:47.639148 (+ 43us) server_negotiation.cc:407] Connection header received
0811 12:36:47.640863 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:47.640890 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:47.641312 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:47.641645 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:47.643535 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:47.644940 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:47.646861 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:47.647824 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:47.647958 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:47.652756 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:47.652766 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:47.652786 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:47.652838 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:47.656863 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:47.657601 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:47.657610 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:47.657624 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:47.657885 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:47.658228 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:47.658233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:47.658236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:47.659050 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:47.659357 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:47.659771 (+ 414us) server_negotiation.cc:299] Negotiation successful
0811 12:36:47.660267 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":382,"threads_started":1}
I20260811 12:36:47.661739 28225 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:36:47.661887 28225 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:36:47.662083 28225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:47.664551 28225 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:47.665037 28225 leader_election.cc:302] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad60ebf94bc94228a8b1a1c4fd255ea8; no voters:
I20260811 12:36:47.665624 28225 leader_election.cc:288] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:36:47.665748 28235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:36:47.666392 28235 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 1 LEADER]: Becoming Leader. State: Replica: ad60ebf94bc94228a8b1a1c4fd255ea8, State: Running, Role: LEADER
I20260811 12:36:47.666934 28235 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:47.667550 28225 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:47.669623 28240 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad60ebf94bc94228a8b1a1c4fd255ea8. Latest consensus state: current_term: 1 leader_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } } }
I20260811 12:36:47.670045 28240 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:47.670547 28238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } } }
I20260811 12:36:47.670855 28238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:47.672336 28248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:36:47.677549 28248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:36:47.687017 28248 catalog_manager.cc:1347] Generated new cluster ID: 0e787c31455c4a4abc11763297f5e54e
I20260811 12:36:47.687222 28248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:36:47.717556 28248 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:36:47.718941 28248 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:36:47.734146 28248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Generated new TSK 0
I20260811 12:36:47.735082 28248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:36:47.813974 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:46081
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:48.165027 28266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:48.165400 28266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:48.165524 28266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:48.173213 28266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:48.173346 28266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:48.173471 28266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:36:48.184011 28266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:48.185436 28266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:48.186923 28266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:48.200791 28279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:48.200791 28278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:48.200748 28281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:48.201655 28266 server_base.cc:1034] running on GCE node
I20260811 12:36:48.202286 28266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:48.203248 28266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:48.204473 28266 hybrid_clock.cc:648] HybridClock initialized: now 1786451808204412 us; error 48 us; skew 500 ppm
I20260811 12:36:48.204794 28266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:48 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451808, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:48.207988 28266 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:36:48.209553 28266 webserver.cc:460] Webserver started at http://127.3.204.65:39215/ using document root <none> and password file <none>
I20260811 12:36:48.210278 28266 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:48.210397 28266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:48.210706 28266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:36:48.213788 28266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "44d2f7a3681e4e53af87f8426cfd4977"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "4aa95f85ce8aa05f7d5785ff5fce5f51"
server_key_iv: "ccd563cd9a264bc8c72ee01fd327c826"
server_key_version: "encryptionkey@0"
I20260811 12:36:48.214569 28266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "44d2f7a3681e4e53af87f8426cfd4977"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "4aa95f85ce8aa05f7d5785ff5fce5f51"
server_key_iv: "ccd563cd9a264bc8c72ee01fd327c826"
server_key_version: "encryptionkey@0"
I20260811 12:36:48.220181 28266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:36:48.223790 28291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:48.225261 28266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:36:48.225473 28266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "44d2f7a3681e4e53af87f8426cfd4977"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "4aa95f85ce8aa05f7d5785ff5fce5f51"
server_key_iv: "ccd563cd9a264bc8c72ee01fd327c826"
server_key_version: "encryptionkey@0"
I20260811 12:36:48.225684 28266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:48.244123 28266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:48.248708 28266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:48.249038 28266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:48.250120 28266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:48.251657 28266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:48.251748 28266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:48.251892 28266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:48.251978 28266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:48.286926 28266 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:44595
I20260811 12:36:48.286964 28507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:44595 every 8 connection(s)
I20260811 12:36:48.288375 28266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:36:48.298115 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 28266
I20260811 12:36:48.298321 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:36:48.299053 3889 external_mini_cluster.cc:1428] Setting key 608375afe4a08a75577dafd575e4757b
Aug 11 12:36:48 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451808, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:48.308588 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:48.291007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:43755 (local address 127.3.204.126:46081)
0811 12:36:48.291466 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:48.291471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:48.292455 (+ 984us) server_negotiation.cc:407] Connection header received
0811 12:36:48.294292 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:48.294298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:48.294382 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:48.294481 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:48.296015 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:48.296778 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:48.298394 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:48.298830 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:48.298954 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:48.302779 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:48.302784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:48.302787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:48.302827 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:48.305457 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:48.306196 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:48.306200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:48.306203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:48.306325 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:48.306775 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:48.306778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:48.306781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:48.307053 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:48.307195 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:48.307947 (+ 752us) server_negotiation.cc:299] Negotiation successful
0811 12:36:48.308227 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":250,"threads_started":1}
I20260811 12:36:48.309608 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:48.291488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:43755)
0811 12:36:48.292275 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:48.292314 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:48.294015 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:48.294732 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:48.294741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:48.295275 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:48.295843 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:48.295861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:48.296957 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:48.296961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:48.298243 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:48.298260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:48.299065 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:48.299073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:48.299354 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:48.300029 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:48.300052 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:48.302563 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:48.305600 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:48.305610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:48.305629 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:48.306071 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:48.306443 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:48.306448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:48.306452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:48.306662 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:48.307199 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:48.307208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:48.307567 (+ 359us) client_negotiation.cc:769] Sending connection context
0811 12:36:48.307830 (+ 263us) client_negotiation.cc:240] Negotiation successful
0811 12:36:48.308420 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":299,"threads_started":1}
I20260811 12:36:48.311264 28510 heartbeater.cc:347] Connected to a master server at 127.3.204.126:46081
I20260811 12:36:48.311596 28510 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:48.312378 28510 heartbeater.cc:511] Master 127.3.204.126:46081 requested a full tablet report, sending...
I20260811 12:36:48.314436 28156 ts_manager.cc:194] Registered new tserver with Master: 44d2f7a3681e4e53af87f8426cfd4977 (127.3.204.65:44595)
I20260811 12:36:48.316191 28156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:43755
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:36:48.360006 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:46081
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:48.698356 28524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:48.698686 28524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:48.698781 28524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:48.705968 28524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:48.706086 28524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:48.706202 28524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:36:48.716033 28524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:48.717502 28524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:48.718851 28524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:48.732378 28532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:48.732339 28531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:48.732419 28536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:48.733340 28524 server_base.cc:1034] running on GCE node
I20260811 12:36:48.734007 28524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:48.735117 28524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:48.736317 28524 hybrid_clock.cc:648] HybridClock initialized: now 1786451808736262 us; error 52 us; skew 500 ppm
I20260811 12:36:48.736645 28524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:48 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451808, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:48.740202 28524 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:36:48.741662 28524 webserver.cc:460] Webserver started at http://127.3.204.66:40135/ using document root <none> and password file <none>
I20260811 12:36:48.742376 28524 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:48.742491 28524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:48.742753 28524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:36:48.745810 28524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "dde40baade934b509310a4ccc878cc4b"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "d238b65a13f82c692a35b9f279ebaab1"
server_key_iv: "71440b8e7fdc118a7dfe796eef125b4b"
server_key_version: "encryptionkey@0"
I20260811 12:36:48.746572 28524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "dde40baade934b509310a4ccc878cc4b"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "d238b65a13f82c692a35b9f279ebaab1"
server_key_iv: "71440b8e7fdc118a7dfe796eef125b4b"
server_key_version: "encryptionkey@0"
I20260811 12:36:48.752216 28524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:36:48.756155 28548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:48.757576 28524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:36:48.757730 28524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "dde40baade934b509310a4ccc878cc4b"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "d238b65a13f82c692a35b9f279ebaab1"
server_key_iv: "71440b8e7fdc118a7dfe796eef125b4b"
server_key_version: "encryptionkey@0"
I20260811 12:36:48.757917 28524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:48.791437 28524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:48.795997 28524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:48.796290 28524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:48.797315 28524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:48.798920 28524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:48.799003 28524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:48.799100 28524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:48.799144 28524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:48.830591 28524 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:41589
I20260811 12:36:48.830634 28751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:41589 every 8 connection(s)
I20260811 12:36:48.831879 28524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:36:48.839027 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 28524
I20260811 12:36:48.839200 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:36:48.839658 3889 external_mini_cluster.cc:1428] Setting key f8129c7039d20643001f93d853c1809b
Aug 11 12:36:48 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451808, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:48.853259 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:48.834440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:45669 (local address 127.3.204.126:46081)
0811 12:36:48.834906 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:48.834911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:48.835732 (+ 821us) server_negotiation.cc:407] Connection header received
0811 12:36:48.837651 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:48.837657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:48.837743 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:48.837846 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:48.839542 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:48.840405 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:48.842327 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:48.842762 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:48.842865 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:48.846925 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:48.846931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:48.846934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:48.846977 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:48.849527 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:48.850408 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:48.850414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:48.850417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:48.850605 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:48.851239 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:48.851244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:48.851247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:48.851608 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:48.851786 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:48.852645 (+ 859us) server_negotiation.cc:299] Negotiation successful
0811 12:36:48.852895 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":225,"threads_started":1}
I20260811 12:36:48.854444 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:48.834848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:45669)
0811 12:36:48.835552 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:48.835595 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:48.837343 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:48.838130 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:48.838140 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:48.838696 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:48.839358 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:48.839379 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:48.840579 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:48.840585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:48.842176 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:48.842192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:48.843002 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:48.843010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:48.843306 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:48.844072 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:48.844130 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:48.846666 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:48.849677 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:48.849687 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:48.849706 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:48.850266 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:48.850739 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:48.850743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:48.850746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:48.851111 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:48.851756 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:48.851765 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:48.852226 (+ 461us) client_negotiation.cc:769] Sending connection context
0811 12:36:48.852540 (+ 314us) client_negotiation.cc:240] Negotiation successful
0811 12:36:48.853045 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":275,"threads_started":1}
I20260811 12:36:48.855932 28752 heartbeater.cc:347] Connected to a master server at 127.3.204.126:46081
I20260811 12:36:48.856266 28752 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:48.857017 28752 heartbeater.cc:511] Master 127.3.204.126:46081 requested a full tablet report, sending...
I20260811 12:36:48.858512 28156 ts_manager.cc:194] Registered new tserver with Master: dde40baade934b509310a4ccc878cc4b (127.3.204.66:41589)
I20260811 12:36:48.859338 28156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:45669
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:36:48.901459 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:46081
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:49.245504 28767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:49.245795 28767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:49.245926 28767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:49.253417 28767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:49.253551 28767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:49.253705 28767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:36:49.264073 28767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:49.265434 28767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:49.266762 28767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:49.279987 28786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:49.280107 28779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:49.280015 28783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:49.280905 28767 server_base.cc:1034] running on GCE node
I20260811 12:36:49.281589 28767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:49.282591 28767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:49.283826 28767 hybrid_clock.cc:648] HybridClock initialized: now 1786451809283774 us; error 42 us; skew 500 ppm
I20260811 12:36:49.284184 28767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:49 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451809, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:49.287739 28767 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:36:49.289500 28767 webserver.cc:460] Webserver started at http://127.3.204.67:39313/ using document root <none> and password file <none>
I20260811 12:36:49.290211 28767 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:49.290334 28767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:49.290664 28767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:36:49.293668 28767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b"
format_stamp: "Formatted at 2026-08-11 12:36:49 on dist-test-slave-lgr5"
server_key: "f987af3b6391c6872a89a5ce2f54d48d"
server_key_iv: "a090f6157c622cc3f51db87ccc3239e3"
server_key_version: "encryptionkey@0"
I20260811 12:36:49.294451 28767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b"
format_stamp: "Formatted at 2026-08-11 12:36:49 on dist-test-slave-lgr5"
server_key: "f987af3b6391c6872a89a5ce2f54d48d"
server_key_iv: "a090f6157c622cc3f51db87ccc3239e3"
server_key_version: "encryptionkey@0"
I20260811 12:36:49.299926 28767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:36:49.303591 28801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:49.305161 28767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:36:49.305341 28767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b"
format_stamp: "Formatted at 2026-08-11 12:36:49 on dist-test-slave-lgr5"
server_key: "f987af3b6391c6872a89a5ce2f54d48d"
server_key_iv: "a090f6157c622cc3f51db87ccc3239e3"
server_key_version: "encryptionkey@0"
I20260811 12:36:49.305546 28767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:49.319347 28510 heartbeater.cc:503] Master 127.3.204.126:46081 was elected leader, sending a full tablet report...
I20260811 12:36:49.332634 28767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:49.337085 28767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:49.337375 28767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:49.338366 28767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:49.339854 28767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:49.339948 28767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:49.340075 28767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:49.340169 28767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260811 12:36:49.371156 28767 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:45173
I20260811 12:36:49.371198 28980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:45173 every 8 connection(s)
I20260811 12:36:49.372562 28767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:36:49.379632 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 28767
I20260811 12:36:49.379844 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:36:49.380483 3889 external_mini_cluster.cc:1428] Setting key d3ad851149bbecad00a38fe4057efea7
Aug 11 12:36:49 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451809, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:49.393447 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.375193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:34647 (local address 127.3.204.126:46081)
0811 12:36:49.375689 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.375694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.376567 (+ 873us) server_negotiation.cc:407] Connection header received
0811 12:36:49.378443 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.378449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.378543 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.378644 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:49.380702 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.381754 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.383740 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.384217 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.384317 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.387681 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:49.387686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:49.387689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:49.387726 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:49.390218 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.391049 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.391054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.391056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.391199 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.391716 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.391720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.391722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.392026 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:49.392193 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.392987 (+ 794us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.393174 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260811 12:36:49.394513 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.375635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:34647)
0811 12:36:49.376382 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.376434 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:49.378153 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:49.378907 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:49.378915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:49.379596 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:49.380468 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.380498 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.381928 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.381934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.383604 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.383617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.384421 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.384427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.384636 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.385252 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:49.385275 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:49.387455 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:49.390406 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:49.390415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:49.390431 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:49.390896 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:49.391365 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:49.391369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:49.391372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:49.391596 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:49.392202 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:49.392209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:49.392595 (+ 386us) client_negotiation.cc:769] Sending connection context
0811 12:36:49.392852 (+ 257us) client_negotiation.cc:240] Negotiation successful
0811 12:36:49.393274 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":274,"threads_started":1}
I20260811 12:36:49.396039 28982 heartbeater.cc:347] Connected to a master server at 127.3.204.126:46081
I20260811 12:36:49.396395 28982 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:49.397114 28982 heartbeater.cc:511] Master 127.3.204.126:46081 requested a full tablet report, sending...
I20260811 12:36:49.398591 28156 ts_manager.cc:194] Registered new tserver with Master: 9d9e0e0b9ca242b3b0bea47fc26bda7b (127.3.204.67:45173)
I20260811 12:36:49.399405 28156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:34647
I20260811 12:36:49.408810 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:36:49.426559 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.413497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.3.204.126:46081)
0811 12:36:49.413668 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.413672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.414100 (+ 428us) server_negotiation.cc:407] Connection header received
0811 12:36:49.414250 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.414254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.414319 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.414405 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:49.415901 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.416740 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.418191 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.418608 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.418702 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.421251 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:49.421257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:49.421260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:49.421310 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:49.423992 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.424799 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.424803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.424806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.424942 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.425254 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.425257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.425259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.425629 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:49.425820 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.426108 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.426295 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 12:36:49.430689 28156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:36:49.437711 28156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:36:49.464659 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.451333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.3.204.66:41589)
0811 12:36:49.452888 (+ 1555us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.452896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.454881 (+ 1985us) server_negotiation.cc:407] Connection header received
0811 12:36:49.455021 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.455028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.455259 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.455466 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:49.456411 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.458139 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.461157 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.462804 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.462945 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.463208 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.464019 (+ 811us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.464261 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1408,"thread_start_us":714,"threads_started":1}
I20260811 12:36:49.465487 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.451554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:41589 (local address 127.0.0.1:60386)
0811 12:36:49.454584 (+ 3030us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.454604 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:49.454737 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:49.455732 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:49.455738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:49.455779 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:49.456190 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.456208 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.458329 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.458335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.460949 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.460966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.463373 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.463380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.463672 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.463733 (+ 61us) client_negotiation.cc:769] Sending connection context
0811 12:36:49.463892 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 12:36:49.464133 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2907,"mutex_wait_us":29,"thread_start_us":2742,"threads_started":1}
I20260811 12:36:49.468269 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.450863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:45173 (local address 127.0.0.1:35330)
0811 12:36:49.453608 (+ 2745us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.453646 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:49.453835 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:49.454726 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:49.454735 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:49.454769 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:49.455435 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.455455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.461694 (+ 6239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.461700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.465236 (+ 3536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.465254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.467324 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.467344 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.467535 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.467571 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:36:49.467644 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 12:36:49.467828 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2512,"mutex_wait_us":696,"thread_start_us":2349,"threads_started":1}
I20260811 12:36:49.469709 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.451233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.3.204.67:45173)
0811 12:36:49.452639 (+ 1406us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.452649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.454056 (+ 1407us) server_negotiation.cc:407] Connection header received
0811 12:36:49.454185 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.454194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.454469 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.455687 (+ 1218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:49.455999 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.461518 (+ 5519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.465435 (+ 3917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.467089 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.468852 (+ 1763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.469125 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.469214 (+ 89us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.469385 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":1007,"threads_started":1}
I20260811 12:36:49.470379 28638 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9b095e29f24498a74242d4665023c6 (DEFAULT_TABLE table=test-table [id=99f1ceab005945ecb26ab60e6d849ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:49.472218 28638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9b095e29f24498a74242d4665023c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:49.473810 28870 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9b095e29f24498a74242d4665023c6 (DEFAULT_TABLE table=test-table [id=99f1ceab005945ecb26ab60e6d849ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:49.475488 28870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9b095e29f24498a74242d4665023c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:49.477882 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.454399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:53064)
0811 12:36:49.465793 (+ 11394us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.465815 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:49.465927 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:49.471718 (+ 5791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:49.471723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:49.471753 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:49.471959 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.471991 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.473628 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.473633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.475771 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.475784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.477127 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.477133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.477311 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.477345 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 12:36:49.477437 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 12:36:49.477587 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11263,"thread_start_us":11758,"threads_started":1}
I20260811 12:36:49.477937 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.454946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.3.204.65:44595)
0811 12:36:49.471079 (+ 16133us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.471089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.471120 (+ 31us) server_negotiation.cc:407] Connection header received
0811 12:36:49.471236 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.471244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.471512 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.471716 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:49.472255 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.473490 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.475927 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.476951 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.477084 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.477339 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.477480 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.477642 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15931,"thread_start_us":16777,"threads_started":1}
I20260811 12:36:49.481669 28391 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9b095e29f24498a74242d4665023c6 (DEFAULT_TABLE table=test-table [id=99f1ceab005945ecb26ab60e6d849ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:49.482963 28391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9b095e29f24498a74242d4665023c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:49.487668 29015 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Bootstrap starting.
I20260811 12:36:49.490171 29016 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Bootstrap starting.
I20260811 12:36:49.491518 29015 tablet_bootstrap.cc:654] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:49.492944 29015 log.cc:824] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:49.493180 29016 tablet_bootstrap.cc:654] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:49.494441 29016 log.cc:824] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:49.497370 29015 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: No bootstrap required, opened a new log
I20260811 12:36:49.497673 29016 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: No bootstrap required, opened a new log
I20260811 12:36:49.497665 29015 ts_tablet_manager.cc:1397] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260811 12:36:49.498064 29016 ts_tablet_manager.cc:1397] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260811 12:36:49.498219 29024 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Bootstrap starting.
I20260811 12:36:49.502000 29024 tablet_bootstrap.cc:654] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:49.503392 29024 log.cc:824] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:49.506527 29016 raft_consensus.cc:348] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.506829 29024 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: No bootstrap required, opened a new log
I20260811 12:36:49.506935 29016 raft_consensus.cc:374] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:49.507071 29016 raft_consensus.cc:729] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d9e0e0b9ca242b3b0bea47fc26bda7b, State: Initialized, Role: FOLLOWER
I20260811 12:36:49.507120 29024 ts_tablet_manager.cc:1397] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260811 12:36:49.507691 29016 consensus_queue.cc:260] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.507650 29015 raft_consensus.cc:348] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.508181 29015 raft_consensus.cc:374] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:49.508324 29015 raft_consensus.cc:729] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde40baade934b509310a4ccc878cc4b, State: Initialized, Role: FOLLOWER
I20260811 12:36:49.508889 28982 heartbeater.cc:503] Master 127.3.204.126:46081 was elected leader, sending a full tablet report...
I20260811 12:36:49.509173 29015 consensus_queue.cc:260] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.509579 29016 ts_tablet_manager.cc:1428] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Time spent starting tablet: real 0.011s user 0.000s sys 0.009s
I20260811 12:36:49.510471 28752 heartbeater.cc:503] Master 127.3.204.126:46081 was elected leader, sending a full tablet report...
I20260811 12:36:49.511180 29015 ts_tablet_manager.cc:1428] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260811 12:36:49.516683 29024 raft_consensus.cc:348] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.517135 29024 raft_consensus.cc:374] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:49.517226 29024 raft_consensus.cc:729] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d2f7a3681e4e53af87f8426cfd4977, State: Initialized, Role: FOLLOWER
I20260811 12:36:49.517728 29024 consensus_queue.cc:260] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.519529 29024 ts_tablet_manager.cc:1428] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260811 12:36:49.540829 28514 tablet.cc:2369] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:36:49.565502 29031 raft_consensus.cc:482] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:36:49.565858 29031 raft_consensus.cc:504] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.567786 29031 leader_election.cc:288] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d9e0e0b9ca242b3b0bea47fc26bda7b (127.3.204.67:45173), dde40baade934b509310a4ccc878cc4b (127.3.204.66:41589)
I20260811 12:36:49.575147 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.568277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:58343 (local address 127.3.204.67:45173)
0811 12:36:49.568506 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.568514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.569243 (+ 729us) server_negotiation.cc:407] Connection header received
0811 12:36:49.569468 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.569472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.569541 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.569626 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:49.570285 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.571101 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.573273 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.574109 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.574237 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.574313 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.574751 (+ 438us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.574877 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:36:49.575269 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.568179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:41589 (local address 127.3.204.65:59247)
0811 12:36:49.569397 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.569412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:49.569546 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:49.569872 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:49.569876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:49.569905 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:49.570294 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.570302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.571284 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.571288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.573067 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.573083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.574407 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.574413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.574544 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.574592 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 12:36:49.574668 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 12:36:49.574797 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":72,"thread_start_us":844,"threads_started":1}
I20260811 12:36:49.575320 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.568012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:45173 (local address 127.3.204.65:58343)
0811 12:36:49.569099 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.569121 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:49.569309 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:49.569658 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:49.569663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:49.569690 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:49.570104 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.570121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.571244 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.571251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.573104 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:49.573123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.574351 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.574359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:49.574582 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.574607 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 12:36:49.574705 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 12:36:49.574848 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":54,"thread_start_us":676,"threads_started":1}
I20260811 12:36:49.575763 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.568428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:59247 (local address 127.3.204.66:41589)
0811 12:36:49.568711 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.568718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.569509 (+ 791us) server_negotiation.cc:407] Connection header received
0811 12:36:49.569681 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.569685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.569747 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.569824 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:49.570447 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.571146 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.573245 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.574162 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.574372 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.574458 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.574730 (+ 272us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.574874 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260811 12:36:49.576925 28676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9b095e29f24498a74242d4665023c6" candidate_uuid: "44d2f7a3681e4e53af87f8426cfd4977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde40baade934b509310a4ccc878cc4b" is_pre_election: true
I20260811 12:36:49.576934 28902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9b095e29f24498a74242d4665023c6" candidate_uuid: "44d2f7a3681e4e53af87f8426cfd4977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" is_pre_election: true
I20260811 12:36:49.577586 28902 raft_consensus.cc:2454] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d2f7a3681e4e53af87f8426cfd4977 in term 0.
I20260811 12:36:49.577586 28676 raft_consensus.cc:2454] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d2f7a3681e4e53af87f8426cfd4977 in term 0.
I20260811 12:36:49.578425 28298 leader_election.cc:302] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d2f7a3681e4e53af87f8426cfd4977, dde40baade934b509310a4ccc878cc4b; no voters:
I20260811 12:36:49.578892 29031 raft_consensus.cc:2781] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:36:49.579056 29031 raft_consensus.cc:482] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:36:49.579157 29031 raft_consensus.cc:3037] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:49.581451 29031 raft_consensus.cc:504] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.582056 29031 leader_election.cc:288] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [CANDIDATE]: Term 1 election: Requested vote from peers 9d9e0e0b9ca242b3b0bea47fc26bda7b (127.3.204.67:45173), dde40baade934b509310a4ccc878cc4b (127.3.204.66:41589)
I20260811 12:36:49.582571 28676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9b095e29f24498a74242d4665023c6" candidate_uuid: "44d2f7a3681e4e53af87f8426cfd4977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde40baade934b509310a4ccc878cc4b"
I20260811 12:36:49.582588 28902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9b095e29f24498a74242d4665023c6" candidate_uuid: "44d2f7a3681e4e53af87f8426cfd4977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b"
I20260811 12:36:49.582768 28676 raft_consensus.cc:3037] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:49.582785 28902 raft_consensus.cc:3037] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Advancing to term 1
W20260811 12:36:49.583889 28754 tablet.cc:2369] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:36:49.585247 28902 raft_consensus.cc:2454] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d2f7a3681e4e53af87f8426cfd4977 in term 1.
I20260811 12:36:49.585644 28676 raft_consensus.cc:2454] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d2f7a3681e4e53af87f8426cfd4977 in term 1.
I20260811 12:36:49.585645 28294 leader_election.cc:302] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d2f7a3681e4e53af87f8426cfd4977, 9d9e0e0b9ca242b3b0bea47fc26bda7b; no voters:
I20260811 12:36:49.585928 29031 raft_consensus.cc:2781] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:36:49.586400 29031 raft_consensus.cc:686] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 LEADER]: Becoming Leader. State: Replica: 44d2f7a3681e4e53af87f8426cfd4977, State: Running, Role: LEADER
I20260811 12:36:49.586861 29031 consensus_queue.cc:237] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:49.591310 28156 catalog_manager.cc:5495] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d2f7a3681e4e53af87f8426cfd4977 (127.3.204.65). New cstate: current_term: 1 leader_uuid: "44d2f7a3681e4e53af87f8426cfd4977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } health_report { overall_health: HEALTHY } } }
W20260811 12:36:49.624318 28985 tablet.cc:2369] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:36:49.632205 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.625676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.3.204.65:44595)
0811 12:36:49.625885 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.625891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.625912 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:36:49.626016 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.626021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.626133 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.626283 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:36:49.626685 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.627745 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.629316 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.630041 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.630170 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.630757 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:36:49.631520 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:36:49.631663 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.631812 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.631905 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:36:49.641871 28902 raft_consensus.cc:1261] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 1 FOLLOWER]: Refusing update from remote peer 44d2f7a3681e4e53af87f8426cfd4977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:36:49.641872 28676 raft_consensus.cc:1261] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 1 FOLLOWER]: Refusing update from remote peer 44d2f7a3681e4e53af87f8426cfd4977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:36:49.642936 29034 consensus_queue.cc:1035] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:36:49.643364 29031 consensus_queue.cc:1035] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:36:49.657545 29040 mvcc.cc:205] Tried to move back new op lower bound from 7317306612282343424 to 7317306612071510016. Current Snapshot: MvccSnapshot[applied={T|T < 7317306612282343424}]
I20260811 12:36:49.672408 28155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:36:49.672758 28155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:36:49.676973 28155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Sending DeleteTablet for 3 replicas of tablet 2a9b095e29f24498a74242d4665023c6
I20260811 12:36:49.678411 28391 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9b095e29f24498a74242d4665023c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:49 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:53064
I20260811 12:36:49.678417 28638 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9b095e29f24498a74242d4665023c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:49 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:60386
I20260811 12:36:49.678472 28870 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9b095e29f24498a74242d4665023c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:49 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:35330
I20260811 12:36:49.682082 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 28266
I20260811 12:36:49.695209 29059 tablet_replica.cc:330] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: stopping tablet replica
I20260811 12:36:49.695772 29059 raft_consensus.cc:2229] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:36:49.696384 29059 raft_consensus.cc:2258] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:49.699115 29059 ts_tablet_manager.cc:1905] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 12:36:49 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:36:49.704742 29060 tablet_replica.cc:330] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: stopping tablet replica
I20260811 12:36:49.705309 29060 raft_consensus.cc:2229] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:36:49.705914 29060 raft_consensus.cc:2258] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:49.706800 29057 tablet_replica.cc:330] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: stopping tablet replica
I20260811 12:36:49.707253 29057 raft_consensus.cc:2229] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:36:49.708038 29057 raft_consensus.cc:2258] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:49.709906 29059 ts_tablet_manager.cc:1918] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:49.710078 29059 log.cc:1196] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/2a9b095e29f24498a74242d4665023c6
I20260811 12:36:49.710327 29060 ts_tablet_manager.cc:1905] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:49.710599 29059 ts_tablet_manager.cc:1939] T 2a9b095e29f24498a74242d4665023c6 P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Deleting consensus metadata
I20260811 12:36:49.713344 28120 catalog_manager.cc:4841] TS 9d9e0e0b9ca242b3b0bea47fc26bda7b (127.3.204.67:45173): tablet 2a9b095e29f24498a74242d4665023c6 (table test-table [id=99f1ceab005945ecb26ab60e6d849ea6]) successfully deleted
I20260811 12:36:49.715817 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.683028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.3.204.65:44595)
0811 12:36:49.683225 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.683233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.687419 (+ 4186us) server_negotiation.cc:407] Connection header received
0811 12:36:49.688521 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.688528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.688661 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.688751 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:49.693201 (+ 4450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.694578 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.698487 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.699260 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.699365 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.706124 (+ 6759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:49.706144 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:49.706167 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:49.706221 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:49.711398 (+ 5177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.712407 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.712416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.712420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.712611 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.713142 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.713147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.713150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.713624 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:49.715010 (+ 1386us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.715172 (+ 162us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.715436 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 12:36:49.716629 28503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:49.717484 29060 ts_tablet_manager.cc:1918] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:49.717669 29060 log.cc:1196] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/2a9b095e29f24498a74242d4665023c6
I20260811 12:36:49.718227 29060 ts_tablet_manager.cc:1939] T 2a9b095e29f24498a74242d4665023c6 P dde40baade934b509310a4ccc878cc4b: Deleting consensus metadata
I20260811 12:36:49.720192 28128 catalog_manager.cc:4841] TS dde40baade934b509310a4ccc878cc4b (127.3.204.66:41589): tablet 2a9b095e29f24498a74242d4665023c6 (table test-table [id=99f1ceab005945ecb26ab60e6d849ea6]) successfully deleted
I20260811 12:36:49.815871 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 28266
I20260811 12:36:49.840857 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 28524
W20260811 12:36:49.841050 28125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.65:44595 (error 108)
I20260811 12:36:49.843345 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.842784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52634)
0811 12:36:49.843022 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.843152 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260811 12:36:49.843657 28125 catalog_manager.cc:4584] TS 44d2f7a3681e4e53af87f8426cfd4977 (127.3.204.65:44595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a9b095e29f24498a74242d4665023c6: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Aug 11 12:36:49 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:36:49.857728 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.841745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.3.204.66:41589)
0811 12:36:49.841958 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.841965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.841987 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:36:49.842057 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.842062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.842144 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.842258 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:49.843737 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.844720 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.846525 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.846980 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.847120 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.851910 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:49.851925 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:49.851941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:49.851986 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:49.854922 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.855665 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.855672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.855674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.855814 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.856275 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.856278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.856280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.856737 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:49.856970 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.857195 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.857377 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 12:36:49.858461 28746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:49.897500 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.897025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52642)
0811 12:36:49.897246 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.897351 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260811 12:36:49.898685 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.898285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52646)
0811 12:36:49.898448 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.898528 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 12:36:49.948908 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 28524
I20260811 12:36:49.959187 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.958591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52652)
0811 12:36:49.958847 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.958980 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260811 12:36:49.960316 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.959918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52666)
0811 12:36:49.960055 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:49.960161 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 12:36:49.979379 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 28767
Aug 11 12:36:49 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:36:49.994794 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:49.980258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.3.204.67:45173)
0811 12:36:49.980446 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:49.980451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:49.980513 (+ 62us) server_negotiation.cc:407] Connection header received
0811 12:36:49.980734 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:49.980740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:49.980830 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:49.980917 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:49.982329 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.983123 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.984626 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:49.985063 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:49.985166 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:49.989576 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:49.989592 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:49.989611 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:49.989648 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:49.992516 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.993160 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.993166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.993169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.993307 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:49.993592 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:49.993595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:49.993597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:49.993929 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:49.994140 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:49.994363 (+ 223us) server_negotiation.cc:299] Negotiation successful
0811 12:36:49.994555 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 12:36:49.995450 28969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:50.040825 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:50.040301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52682)
0811 12:36:50.040532 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:50.040643 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 12:36:50.041929 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:50.041539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52694)
0811 12:36:50.041725 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:50.041803 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260811 12:36:50.085368 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 28767
I20260811 12:36:50.115769 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 28096
I20260811 12:36:50.117211 28218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:50.229359 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:50.228621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52700)
0811 12:36:50.228998 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:50.229132 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
I20260811 12:36:50.229372 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 28096
I20260811 12:36:50.258352 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46081
--webserver_interface=127.3.204.126
--webserver_port=46727
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:46081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:50.610615 29069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:50.610931 29069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:50.611068 29069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:50.618434 29069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:50.618561 29069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:50.618633 29069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:50.618713 29069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:50.618774 29069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:50.628904 29069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:46081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=46727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:50.630291 29069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:50.631640 29069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:50.655649 29087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:50.655757 29089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:50.655676 29095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:50.656771 29069 server_base.cc:1034] running on GCE node
I20260811 12:36:50.657732 29069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:50.659397 29069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:50.660640 29069 hybrid_clock.cc:648] HybridClock initialized: now 1786451810660579 us; error 34 us; skew 500 ppm
I20260811 12:36:50.661073 29069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:50 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451810, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:50.664793 29069 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:36:50.666553 29069 webserver.cc:460] Webserver started at http://127.3.204.126:46727/ using document root <none> and password file <none>
I20260811 12:36:50.667294 29069 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:50.667431 29069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:50.673666 29069 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260811 12:36:50.676699 29108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:50.678190 29069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:36:50.678400 29069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8"
format_stamp: "Formatted at 2026-08-11 12:36:47 on dist-test-slave-lgr5"
server_key: "12538c978bf6a3fc857344254f6fce75"
server_key_iv: "84a3f6302c8210fcbef72b1369a1215a"
server_key_version: "encryptionkey@0"
I20260811 12:36:50.679659 29069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:50.709034 29069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:50.713721 29069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:50.714003 29069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:50.733309 29069 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:46081
I20260811 12:36:50.733330 29223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:46081 every 8 connection(s)
I20260811 12:36:50.734969 29069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:36:50.737650 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 29069
I20260811 12:36:50.738667 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:44595
--local_ip_for_outbound_sockets=127.3.204.65
--tserver_master_addrs=127.3.204.126:46081
--webserver_port=39215
--webserver_interface=127.3.204.65
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:36:50.744974 29226 sys_catalog.cc:263] Verifying existing consensus state
I20260811 12:36:50.749768 29226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Bootstrap starting.
I20260811 12:36:50.771140 29226 log.cc:824] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:50.784018 29226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:36:50.784636 29226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Bootstrap complete.
I20260811 12:36:50.793862 29226 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:50.794744 29226 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad60ebf94bc94228a8b1a1c4fd255ea8, State: Initialized, Role: FOLLOWER
I20260811 12:36:50.795380 29226 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:50.795624 29226 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:36:50.795737 29226 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:36:50.795902 29226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 1 FOLLOWER]: Advancing to term 2
I20260811 12:36:50.799897 29226 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:50.800455 29226 leader_election.cc:302] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad60ebf94bc94228a8b1a1c4fd255ea8; no voters:
I20260811 12:36:50.801059 29226 leader_election.cc:288] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 12:36:50.801203 29237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 12:36:50.802342 29237 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 2 LEADER]: Becoming Leader. State: Replica: ad60ebf94bc94228a8b1a1c4fd255ea8, State: Running, Role: LEADER
I20260811 12:36:50.802929 29237 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:50.803399 29226 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:50.805558 29243 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad60ebf94bc94228a8b1a1c4fd255ea8. Latest consensus state: current_term: 2 leader_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } } }
I20260811 12:36:50.805583 29242 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } } }
I20260811 12:36:50.806008 29243 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:50.806005 29242 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:50.807876 29250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:36:50.815769 29250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:36:50.817235 29250 catalog_manager.cc:1259] Loaded cluster ID: 0e787c31455c4a4abc11763297f5e54e
I20260811 12:36:50.817342 29250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:36:50.820598 29250 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260811 12:36:50.838490 29250 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260811 12:36:50.848606 29250 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:36:50.850535 29250 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Loaded TSK: 0
I20260811 12:36:50.852934 29250 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 12:36:51.122889 29232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:51.123204 29232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:51.123342 29232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:51.130874 29232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:51.131050 29232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:51.131207 29232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:36:51.141907 29232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:44595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=39215
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:51.143290 29232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:51.144729 29232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:51.159818 29285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:51.159898 29286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:51.160787 29290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:51.161168 29232 server_base.cc:1034] running on GCE node
I20260811 12:36:51.161825 29232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:51.162879 29232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:51.164127 29232 hybrid_clock.cc:648] HybridClock initialized: now 1786451811164039 us; error 55 us; skew 500 ppm
I20260811 12:36:51.164454 29232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:51 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451811, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:51.167575 29232 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:36:51.169247 29232 webserver.cc:460] Webserver started at http://127.3.204.65:39215/ using document root <none> and password file <none>
I20260811 12:36:51.169974 29232 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:51.170106 29232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:51.175748 29232 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 12:36:51.178758 29305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:51.180186 29232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 12:36:51.180438 29232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "44d2f7a3681e4e53af87f8426cfd4977"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "4aa95f85ce8aa05f7d5785ff5fce5f51"
server_key_iv: "ccd563cd9a264bc8c72ee01fd327c826"
server_key_version: "encryptionkey@0"
I20260811 12:36:51.181382 29232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:51.198962 29232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:51.203450 29232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:51.203740 29232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:51.204861 29232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:51.206853 29317 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260811 12:36:51.213357 29232 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 12:36:51.213557 29232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260811 12:36:51.213696 29232 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 12:36:51.217108 29232 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 12:36:51.217206 29232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260811 12:36:51.217645 29317 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Bootstrap starting.
I20260811 12:36:51.227389 29317 log.cc:824] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:51.243403 29317 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:36:51.244320 29317 tablet_bootstrap.cc:492] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Bootstrap complete.
I20260811 12:36:51.244964 29317 ts_tablet_manager.cc:1397] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20260811 12:36:51.254899 29317 raft_consensus.cc:348] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:51.256392 29317 raft_consensus.cc:729] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d2f7a3681e4e53af87f8426cfd4977, State: Initialized, Role: FOLLOWER
I20260811 12:36:51.257241 29317 consensus_queue.cc:260] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:51.259045 29317 ts_tablet_manager.cc:1428] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260811 12:36:51.261459 29232 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:44595
I20260811 12:36:51.261538 29502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:44595 every 8 connection(s)
I20260811 12:36:51.262748 29232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:36:51.263803 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 29232
I20260811 12:36:51.264570 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:41589
--local_ip_for_outbound_sockets=127.3.204.66
--tserver_master_addrs=127.3.204.126:46081
--webserver_port=40135
--webserver_interface=127.3.204.66
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:36:51 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451811, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:51.286826 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:51.266048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:52581)
0811 12:36:51.267050 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:51.267088 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:51.267591 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:51.269065 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:51.269077 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:51.269796 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:51.270430 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:51.270450 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.272119 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.272124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:51.273528 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:51.273543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.274524 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.274531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:51.274809 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:51.275534 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:51.275562 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:51.278069 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:51.281719 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:51.281728 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:51.281741 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:51.282376 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:51.282883 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:51.282887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:51.282889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:51.283103 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:51.283980 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:51.283988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:51.284408 (+ 420us) client_negotiation.cc:769] Sending connection context
0811 12:36:51.284737 (+ 329us) client_negotiation.cc:240] Negotiation successful
0811 12:36:51.285402 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":505,"threads_started":1}
I20260811 12:36:51.286827 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:51.266046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:52581 (local address 127.3.204.126:46081)
0811 12:36:51.266685 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:51.266703 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:51.267235 (+ 532us) server_negotiation.cc:407] Connection header received
0811 12:36:51.268159 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:51.268174 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:51.268534 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:51.268882 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:51.270670 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.271923 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.273678 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.274317 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.274506 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:51.278372 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:51.278381 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:51.278396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:51.278437 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:51.281542 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:51.282551 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:51.282559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:51.282565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:51.282757 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:51.283220 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:51.283223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:51.283226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:51.283824 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:51.284036 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:51.284963 (+ 927us) server_negotiation.cc:299] Negotiation successful
0811 12:36:51.285402 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":258,"threads_started":1}
I20260811 12:36:51.290436 29504 heartbeater.cc:347] Connected to a master server at 127.3.204.126:46081
I20260811 12:36:51.290795 29504 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:51.291553 29504 heartbeater.cc:511] Master 127.3.204.126:46081 requested a full tablet report, sending...
I20260811 12:36:51.294962 29152 ts_manager.cc:194] Registered new tserver with Master: 44d2f7a3681e4e53af87f8426cfd4977 (127.3.204.65:44595)
I20260811 12:36:51.299510 29152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:52581
Aug 11 12:36:51 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451810, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:36:51.319003 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:51.298889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52710)
0811 12:36:51.301437 (+ 2548us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:51.301497 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:51.301696 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:51.302802 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:51.302811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:51.303781 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:51.304175 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:51.304193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.306313 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.306318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:51.307513 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:51.307529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.308686 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.308691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:51.308851 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:51.309504 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:51.309521 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:51.311911 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:51.316054 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:51.316068 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:51.316073 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:51.316555 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:51.317035 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:51.317038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:51.317041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:51.317212 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:51.318042 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:51.318048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:51.318216 (+ 168us) client_negotiation.cc:769] Sending connection context
0811 12:36:51.318378 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 12:36:51.318661 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2238,"thread_start_us":2073,"threads_started":1}
I20260811 12:36:51.319628 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:51.298994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.3.204.65:44595)
0811 12:36:51.302144 (+ 3150us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:51.302154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:51.302189 (+ 35us) server_negotiation.cc:407] Connection header received
0811 12:36:51.302287 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:51.302295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:51.302573 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:51.302736 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:51.304406 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.306135 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.307675 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.308507 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.308633 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:51.312215 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:51.312231 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:51.312246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:51.312295 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:51.315859 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:51.316700 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:51.316709 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:51.316712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:51.316880 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:51.317342 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:51.317347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:51.317350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:51.317852 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:51.318051 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:51.318570 (+ 519us) server_negotiation.cc:299] Negotiation successful
0811 12:36:51.318881 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2967,"thread_start_us":2814,"threads_started":1}
I20260811 12:36:51.321074 29391 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9b095e29f24498a74242d4665023c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:49 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:52710
I20260811 12:36:51.323288 29512 tablet_replica.cc:330] stopping tablet replica
I20260811 12:36:51.323666 29512 raft_consensus.cc:2229] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:36:51.324186 29512 raft_consensus.cc:2258] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:51.325419 29512 ts_tablet_manager.cc:1905] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:51.332602 29512 ts_tablet_manager.cc:1918] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:51.332772 29512 log.cc:1196] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/2a9b095e29f24498a74242d4665023c6
I20260811 12:36:51.333158 29512 ts_tablet_manager.cc:1939] T 2a9b095e29f24498a74242d4665023c6 P 44d2f7a3681e4e53af87f8426cfd4977: Deleting consensus metadata
I20260811 12:36:51.335863 29114 catalog_manager.cc:4841] TS 44d2f7a3681e4e53af87f8426cfd4977 (127.3.204.65:44595): tablet 2a9b095e29f24498a74242d4665023c6 (table test-table [id=99f1ceab005945ecb26ab60e6d849ea6]) successfully deleted
W20260811 12:36:51.618291 29508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:51.618587 29508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:51.618709 29508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:51.625993 29508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:51.626106 29508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:51.626260 29508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:36:51.636325 29508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:41589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=40135
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:51.637737 29508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:51.639070 29508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:51.652140 29517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:51.652185 29519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:51.652191 29524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:51.652926 29508 server_base.cc:1034] running on GCE node
I20260811 12:36:51.653602 29508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:51.654644 29508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:51.655874 29508 hybrid_clock.cc:648] HybridClock initialized: now 1786451811655806 us; error 54 us; skew 500 ppm
I20260811 12:36:51.656289 29508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:51 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451811, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:51.659457 29508 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:36:51.660995 29508 webserver.cc:460] Webserver started at http://127.3.204.66:40135/ using document root <none> and password file <none>
I20260811 12:36:51.661731 29508 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:51.661871 29508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:51.667330 29508 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 12:36:51.670082 29532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:51.671432 29508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:36:51.671591 29508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "dde40baade934b509310a4ccc878cc4b"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "d238b65a13f82c692a35b9f279ebaab1"
server_key_iv: "71440b8e7fdc118a7dfe796eef125b4b"
server_key_version: "encryptionkey@0"
I20260811 12:36:51.672668 29508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:51.699416 29508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:51.703534 29508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:51.703850 29508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:51.705053 29508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:51.706435 29508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:51.706538 29508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:51.706643 29508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:51.706713 29508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:51.738111 29508 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:41589
I20260811 12:36:51.738174 29748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:41589 every 8 connection(s)
I20260811 12:36:51.739593 29508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:36:51.748250 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 29508
I20260811 12:36:51.748952 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:45173
--local_ip_for_outbound_sockets=127.3.204.67
--tserver_master_addrs=127.3.204.126:46081
--webserver_port=39313
--webserver_interface=127.3.204.67
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:36:51 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451811, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:51.762908 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:51.742267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:52567 (local address 127.3.204.126:46081)
0811 12:36:51.742489 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:51.742496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:51.743805 (+ 1309us) server_negotiation.cc:407] Connection header received
0811 12:36:51.745734 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:51.745740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:51.745848 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:51.745976 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:51.747451 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.748629 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.751008 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.751451 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.751563 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:51.755924 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:51.755931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:51.755935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:51.755978 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:51.759048 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:51.759990 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:51.759996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:51.759999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:51.760196 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:51.760724 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:51.760730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:51.760733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:51.761128 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:51.761279 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:51.762307 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0811 12:36:51.762564 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:36:51.763821 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:51.742763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:52567)
0811 12:36:51.743612 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:51.743668 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:51.745447 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:51.746161 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:51.746169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:51.746705 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:51.747275 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:51.747293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.748848 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.748854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:51.750757 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:51.750777 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:51.751696 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:51.751703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:51.751992 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:51.752817 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:51.752848 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:51.755641 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:51.759218 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:51.759228 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:51.759247 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:51.759804 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:51.760331 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:51.760335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:51.760337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:51.760560 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:51.761336 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:51.761345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:51.761849 (+ 504us) client_negotiation.cc:769] Sending connection context
0811 12:36:51.762173 (+ 324us) client_negotiation.cc:240] Negotiation successful
0811 12:36:51.762644 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":334,"threads_started":1}
I20260811 12:36:51.765424 29749 heartbeater.cc:347] Connected to a master server at 127.3.204.126:46081
I20260811 12:36:51.765777 29749 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:51.766644 29749 heartbeater.cc:511] Master 127.3.204.126:46081 requested a full tablet report, sending...
I20260811 12:36:51.768339 29152 ts_manager.cc:194] Registered new tserver with Master: dde40baade934b509310a4ccc878cc4b (127.3.204.66:41589)
I20260811 12:36:51.769207 29152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:52567
W20260811 12:36:52.091751 29753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:52.092111 29753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:52.092243 29753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:52.099520 29753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:52.099661 29753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:52.099828 29753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:36:52.110023 29753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:45173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=39313
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:52.111454 29753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:52.112877 29753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:52.126130 29763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:52.126098 29765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:52.126021 29768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:52.126778 29753 server_base.cc:1034] running on GCE node
I20260811 12:36:52.127382 29753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:52.128383 29753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:52.129581 29753 hybrid_clock.cc:648] HybridClock initialized: now 1786451812129535 us; error 40 us; skew 500 ppm
I20260811 12:36:52.129878 29753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:52 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451812, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:52.133021 29753 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:36:52.134526 29753 webserver.cc:460] Webserver started at http://127.3.204.67:39313/ using document root <none> and password file <none>
I20260811 12:36:52.135231 29753 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:52.135346 29753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:52.140937 29753 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:36:52.143754 29781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:52.145170 29753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:36:52.145315 29753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b"
format_stamp: "Formatted at 2026-08-11 12:36:49 on dist-test-slave-lgr5"
server_key: "f987af3b6391c6872a89a5ce2f54d48d"
server_key_iv: "a090f6157c622cc3f51db87ccc3239e3"
server_key_version: "encryptionkey@0"
I20260811 12:36:52.146281 29753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:52.168731 29753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:52.173096 29753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:52.173334 29753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:52.174355 29753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:52.175837 29753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:52.175927 29753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:52.176019 29753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:52.176075 29753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:52.207186 29753 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:45173
I20260811 12:36:52.207259 29981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:45173 every 8 connection(s)
I20260811 12:36:52.208663 29753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:36:52.212545 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 29753
Aug 11 12:36:52 dist-test-slave-lgr5 krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451812, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:36:52.228839 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.213440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.3.204.126:46081)
0811 12:36:52.213899 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.213905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.214255 (+ 350us) server_negotiation.cc:407] Connection header received
0811 12:36:52.214395 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.214401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.214487 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.214609 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:52.216281 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.217106 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.218631 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.219247 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.219359 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.222869 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:52.222874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:52.222877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:52.222911 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:52.225996 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.226692 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.226696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.226699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.226826 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.227203 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.227207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.227209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.227531 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:52.227798 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.228151 (+ 353us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.228484 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"spinlock_wait_cycles":1408,"thread_start_us":233,"threads_started":1}
I20260811 12:36:52.229641 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.211222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:36803 (local address 127.3.204.126:46081)
0811 12:36:52.211395 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.211400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.212597 (+ 1197us) server_negotiation.cc:407] Connection header received
0811 12:36:52.214744 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.214749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.214826 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.214916 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:52.216562 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.217373 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.219078 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.219601 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.219693 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.223529 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:52.223533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:52.223536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:52.223576 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:52.226378 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.227228 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.227232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.227234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.227382 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.227830 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.227833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.227835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.228143 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:52.228273 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.229112 (+ 839us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.229321 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":3456}
I20260811 12:36:52.230890 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.211706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:36803)
0811 12:36:52.212431 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.212478 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:52.214431 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:52.215162 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:52.215170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:52.215680 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:36:52.216370 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.216399 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.217542 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.217545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.218919 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.218934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.219831 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.219839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.220125 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.220711 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:36:52.220743 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:36:52.223309 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:36:52.226537 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:52.226547 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:52.226566 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:52.227094 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:52.227497 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:36:52.227501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:36:52.227505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:36:52.227711 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:36:52.228265 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:36:52.228273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:36:52.228660 (+ 387us) client_negotiation.cc:769] Sending connection context
0811 12:36:52.228981 (+ 321us) client_negotiation.cc:240] Negotiation successful
0811 12:36:52.229519 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":293,"threads_started":1}
I20260811 12:36:52.232148 29982 heartbeater.cc:347] Connected to a master server at 127.3.204.126:46081
I20260811 12:36:52.232456 29982 heartbeater.cc:464] Registering TS with master...
I20260811 12:36:52.233175 29982 heartbeater.cc:511] Master 127.3.204.126:46081 requested a full tablet report, sending...
I20260811 12:36:52.234572 29152 ts_manager.cc:194] Registered new tserver with Master: 9d9e0e0b9ca242b3b0bea47fc26bda7b (127.3.204.67:45173)
I20260811 12:36:52.235378 29152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:36803
I20260811 12:36:52.244071 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:36:52.251506 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.245179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.3.204.126:46081)
0811 12:36:52.245401 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.245408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.245743 (+ 335us) server_negotiation.cc:407] Connection header received
0811 12:36:52.245968 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.245973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.246114 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.246251 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:36:52.246713 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.247633 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.249220 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.249642 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.249757 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.250343 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:36:52.250882 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:36:52.251029 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.251161 (+ 132us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.251254 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 12:36:52.252664 29152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:36:52.260635 29152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:36:52.275422 29391 tablet_service.cc:1467] Processing CreateTablet for tablet 835eb4017bce4986a7d788c9bb99444a (DEFAULT_TABLE table=test-table [id=de9cbc0e02f04900afce62ff3f8b9e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:52.276607 29391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835eb4017bce4986a7d788c9bb99444a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:52.288975 30010 tablet_bootstrap.cc:492] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Bootstrap starting.
I20260811 12:36:52.291922 30010 tablet_bootstrap.cc:654] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:52.295034 30010 tablet_bootstrap.cc:492] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: No bootstrap required, opened a new log
I20260811 12:36:52.295238 30010 ts_tablet_manager.cc:1397] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 12:36:52.296319 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.273009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:41589 (local address 127.0.0.1:45698)
0811 12:36:52.273705 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.273728 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:52.273867 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:52.287497 (+ 13630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:52.287504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:52.287562 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:52.287967 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.287982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.290677 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.290685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.292963 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.292977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.294807 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.294814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.295022 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.295092 (+ 70us) client_negotiation.cc:769] Sending connection context
0811 12:36:52.295188 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 12:36:52.295355 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":363,"threads_started":1}
I20260811 12:36:52.296150 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.273184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.3.204.66:41589)
0811 12:36:52.286297 (+ 13113us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.286309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.286356 (+ 47us) server_negotiation.cc:407] Connection header received
0811 12:36:52.286506 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.286515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.286858 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.288284 (+ 1426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:52.288663 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.290452 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.293241 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.294564 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.294697 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.294927 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.295531 (+ 604us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.295802 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12883,"thread_start_us":13930,"threads_started":1}
I20260811 12:36:52.296197 30010 raft_consensus.cc:348] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.296651 30010 raft_consensus.cc:374] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:52.296766 30010 raft_consensus.cc:729] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d2f7a3681e4e53af87f8426cfd4977, State: Initialized, Role: FOLLOWER
I20260811 12:36:52.297039 30010 consensus_queue.cc:260] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.297941 30010 ts_tablet_manager.cc:1428] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 12:36:52.298038 29504 heartbeater.cc:503] Master 127.3.204.126:46081 was elected leader, sending a full tablet report...
I20260811 12:36:52.300710 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.274333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:45173 (local address 127.0.0.1:56222)
0811 12:36:52.291640 (+ 17307us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.291665 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:52.291819 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:52.293020 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:52.293025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:52.293056 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:52.293417 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.293433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.295400 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.295407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.298074 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.298092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.299892 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.299899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.300111 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.300146 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:36:52.300242 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 12:36:52.300381 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17157,"thread_start_us":16974,"threads_started":1}
I20260811 12:36:52.301034 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.274793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.3.204.67:45173)
0811 12:36:52.290496 (+ 15703us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.290506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.291980 (+ 1474us) server_negotiation.cc:407] Connection header received
0811 12:36:52.292137 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.292146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.292463 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.292698 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:52.293652 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.295178 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.298268 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.299635 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.299781 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.300059 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.300324 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.300515 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15478,"thread_start_us":15722,"threads_started":1}
I20260811 12:36:52.300467 29624 tablet_service.cc:1467] Processing CreateTablet for tablet 835eb4017bce4986a7d788c9bb99444a (DEFAULT_TABLE table=test-table [id=de9cbc0e02f04900afce62ff3f8b9e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:52.302052 29624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835eb4017bce4986a7d788c9bb99444a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:52.305356 29873 tablet_service.cc:1467] Processing CreateTablet for tablet 835eb4017bce4986a7d788c9bb99444a (DEFAULT_TABLE table=test-table [id=de9cbc0e02f04900afce62ff3f8b9e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:36:52.306922 29873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835eb4017bce4986a7d788c9bb99444a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:36:52.319205 30016 tablet_bootstrap.cc:492] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: Bootstrap starting.
I20260811 12:36:52.322700 30016 tablet_bootstrap.cc:654] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:52.322713 30020 tablet_bootstrap.cc:492] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Bootstrap starting.
I20260811 12:36:52.324188 30016 log.cc:824] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:52.326244 30020 tablet_bootstrap.cc:654] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Neither blocks nor log segments found. Creating new log.
I20260811 12:36:52.327332 30020 log.cc:824] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:52.327483 30016 tablet_bootstrap.cc:492] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: No bootstrap required, opened a new log
I20260811 12:36:52.327770 30016 ts_tablet_manager.cc:1397] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 12:36:52.330075 30020 tablet_bootstrap.cc:492] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: No bootstrap required, opened a new log
I20260811 12:36:52.330327 30020 ts_tablet_manager.cc:1397] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260811 12:36:52.335763 30016 raft_consensus.cc:348] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.336256 30016 raft_consensus.cc:374] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:52.336364 30016 raft_consensus.cc:729] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde40baade934b509310a4ccc878cc4b, State: Initialized, Role: FOLLOWER
I20260811 12:36:52.336995 30016 consensus_queue.cc:260] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.338279 29749 heartbeater.cc:503] Master 127.3.204.126:46081 was elected leader, sending a full tablet report...
I20260811 12:36:52.338310 30020 raft_consensus.cc:348] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.338690 30020 raft_consensus.cc:374] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:36:52.338788 30020 raft_consensus.cc:729] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d9e0e0b9ca242b3b0bea47fc26bda7b, State: Initialized, Role: FOLLOWER
I20260811 12:36:52.338951 30016 ts_tablet_manager.cc:1428] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260811 12:36:52.339375 30020 consensus_queue.cc:260] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.340579 29982 heartbeater.cc:503] Master 127.3.204.126:46081 was elected leader, sending a full tablet report...
I20260811 12:36:52.341255 30020 ts_tablet_manager.cc:1428] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260811 12:36:52.373407 30024 raft_consensus.cc:482] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:36:52.373721 30024 raft_consensus.cc:504] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.375525 30024 leader_election.cc:288] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d9e0e0b9ca242b3b0bea47fc26bda7b (127.3.204.67:45173), 44d2f7a3681e4e53af87f8426cfd4977 (127.3.204.65:44595)
I20260811 12:36:52.382797 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.375828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:50533 (local address 127.3.204.67:45173)
0811 12:36:52.376050 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.376056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.376660 (+ 604us) server_negotiation.cc:407] Connection header received
0811 12:36:52.376963 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.376970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.377057 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.377157 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:52.377755 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.378678 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.380696 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.381731 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.381876 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.381934 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.382394 (+ 460us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.382535 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 12:36:52.382850 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.375708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:45173 (local address 127.3.204.66:50533)
0811 12:36:52.376538 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.376562 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:52.376782 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:52.377186 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:52.377191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:52.377215 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:52.377602 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.377617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.378826 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.378832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.380528 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.380545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.381974 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.381981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.382180 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.382216 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:36:52.382327 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 12:36:52.382479 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":441,"threads_started":1}
I20260811 12:36:52.383492 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.375968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.3.204.66:44627)
0811 12:36:52.376808 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.376824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:36:52.376911 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:36:52.377647 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:36:52.377651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:36:52.377675 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:36:52.378115 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.378129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.379375 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.379379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.380995 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:36:52.381010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.382656 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.382663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:36:52.382866 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.382901 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:36:52.383001 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 12:36:52.383136 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":499,"threads_started":1}
I20260811 12:36:52.383957 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.376684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:44627 (local address 127.3.204.65:44595)
0811 12:36:52.377304 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.377310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.377329 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:36:52.377400 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.377405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.377494 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.377615 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:36:52.378309 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.379211 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.381180 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.382228 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.382404 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.382598 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.383262 (+ 664us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.383453 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":341,"threads_started":1}
I20260811 12:36:52.383952 29902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835eb4017bce4986a7d788c9bb99444a" candidate_uuid: "dde40baade934b509310a4ccc878cc4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" is_pre_election: true
I20260811 12:36:52.384577 29902 raft_consensus.cc:2454] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dde40baade934b509310a4ccc878cc4b in term 0.
I20260811 12:36:52.384829 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835eb4017bce4986a7d788c9bb99444a" candidate_uuid: "dde40baade934b509310a4ccc878cc4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d2f7a3681e4e53af87f8426cfd4977" is_pre_election: true
I20260811 12:36:52.385277 29534 leader_election.cc:302] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d9e0e0b9ca242b3b0bea47fc26bda7b, dde40baade934b509310a4ccc878cc4b; no voters:
I20260811 12:36:52.385360 29430 raft_consensus.cc:2454] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dde40baade934b509310a4ccc878cc4b in term 0.
I20260811 12:36:52.385660 30024 raft_consensus.cc:2781] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:36:52.385848 30024 raft_consensus.cc:482] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:36:52.385987 30024 raft_consensus.cc:3037] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:52.388329 30024 raft_consensus.cc:504] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.388898 30024 leader_election.cc:288] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [CANDIDATE]: Term 1 election: Requested vote from peers 9d9e0e0b9ca242b3b0bea47fc26bda7b (127.3.204.67:45173), 44d2f7a3681e4e53af87f8426cfd4977 (127.3.204.65:44595)
I20260811 12:36:52.389410 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835eb4017bce4986a7d788c9bb99444a" candidate_uuid: "dde40baade934b509310a4ccc878cc4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d2f7a3681e4e53af87f8426cfd4977"
I20260811 12:36:52.389410 29902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835eb4017bce4986a7d788c9bb99444a" candidate_uuid: "dde40baade934b509310a4ccc878cc4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b"
I20260811 12:36:52.389639 29902 raft_consensus.cc:3037] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:52.389638 29430 raft_consensus.cc:3037] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:36:52.392956 29902 raft_consensus.cc:2454] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dde40baade934b509310a4ccc878cc4b in term 1.
I20260811 12:36:52.392993 29430 raft_consensus.cc:2454] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dde40baade934b509310a4ccc878cc4b in term 1.
I20260811 12:36:52.393445 29534 leader_election.cc:302] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 44d2f7a3681e4e53af87f8426cfd4977, 9d9e0e0b9ca242b3b0bea47fc26bda7b, dde40baade934b509310a4ccc878cc4b; no voters:
I20260811 12:36:52.393932 30024 raft_consensus.cc:2781] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:36:52.394528 30024 raft_consensus.cc:686] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 1 LEADER]: Becoming Leader. State: Replica: dde40baade934b509310a4ccc878cc4b, State: Running, Role: LEADER
I20260811 12:36:52.395160 30024 consensus_queue.cc:237] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } }
I20260811 12:36:52.399899 29152 catalog_manager.cc:5495] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b reported cstate change: term changed from 0 to 1, leader changed from <none> to dde40baade934b509310a4ccc878cc4b (127.3.204.66). New cstate: current_term: 1 leader_uuid: "dde40baade934b509310a4ccc878cc4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde40baade934b509310a4ccc878cc4b" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 } health_report { overall_health: UNKNOWN } } }
I20260811 12:36:52.432559 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.427250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.3.204.66:41589)
0811 12:36:52.427469 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.427475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.427497 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:36:52.427599 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.427603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.427686 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.427835 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:36:52.428298 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.429070 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.430525 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.430936 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.431025 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.431476 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:36:52.431984 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:36:52.432136 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.432183 (+ 47us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.432257 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 12:36:52.441555 29902 raft_consensus.cc:1261] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 1 FOLLOWER]: Refusing update from remote peer dde40baade934b509310a4ccc878cc4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:36:52.441555 29430 raft_consensus.cc:1261] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Refusing update from remote peer dde40baade934b509310a4ccc878cc4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:36:52.442679 30031 consensus_queue.cc:1035] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d2f7a3681e4e53af87f8426cfd4977" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 44595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:36:52.443066 30024 consensus_queue.cc:1035] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:36:52.473941 29152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:36:52.474212 29152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:36:52.478296 29152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Sending DeleteTablet for 3 replicas of tablet 835eb4017bce4986a7d788c9bb99444a
I20260811 12:36:52.479394 29624 tablet_service.cc:1514] Processing DeleteTablet for tablet 835eb4017bce4986a7d788c9bb99444a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:52 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:45698
I20260811 12:36:52.479511 29873 tablet_service.cc:1514] Processing DeleteTablet for tablet 835eb4017bce4986a7d788c9bb99444a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:52 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:56222
I20260811 12:36:52.479928 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 29232
I20260811 12:36:52.480137 29391 tablet_service.cc:1514] Processing DeleteTablet for tablet 835eb4017bce4986a7d788c9bb99444a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:36:52 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:52710
I20260811 12:36:52.493943 30057 tablet_replica.cc:330] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: stopping tablet replica
I20260811 12:36:52.494478 30057 raft_consensus.cc:2229] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:36:52.495074 30057 raft_consensus.cc:2258] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:52.495229 30056 tablet_replica.cc:330] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: stopping tablet replica
I20260811 12:36:52.495651 30056 raft_consensus.cc:2229] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:36:52.495816 30058 tablet_replica.cc:330] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: stopping tablet replica
I20260811 12:36:52.496033 30058 raft_consensus.cc:2229] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:36:52.496486 30058 raft_consensus.cc:2258] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:52.496495 30056 raft_consensus.cc:2258] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:36:52.497852 30058 ts_tablet_manager.cc:1905] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:52.498508 30057 ts_tablet_manager.cc:1905] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:52.498582 30056 ts_tablet_manager.cc:1905] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:52.502684 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.480915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.3.204.65:44595)
0811 12:36:52.481201 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.481208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.481231 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:36:52.481296 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.481302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.481385 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.481500 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:52.483485 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.484749 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.487186 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.487943 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.488068 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.493803 (+ 5735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:52.493813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:52.493818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:52.493873 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:52.498401 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.499558 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.499564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.499567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.499747 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.500250 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.500255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.500259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.500604 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:52.500870 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.502084 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.502331 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260811 12:36:52.503605 29495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:52.505362 30056 ts_tablet_manager.cc:1918] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:52.505560 30056 log.cc:1196] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/835eb4017bce4986a7d788c9bb99444a
I20260811 12:36:52.505884 30057 ts_tablet_manager.cc:1918] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:52.506103 30057 log.cc:1196] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/835eb4017bce4986a7d788c9bb99444a
I20260811 12:36:52.506178 30056 ts_tablet_manager.cc:1939] T 835eb4017bce4986a7d788c9bb99444a P dde40baade934b509310a4ccc878cc4b: Deleting consensus metadata
I20260811 12:36:52.506651 30057 ts_tablet_manager.cc:1939] T 835eb4017bce4986a7d788c9bb99444a P 9d9e0e0b9ca242b3b0bea47fc26bda7b: Deleting consensus metadata
I20260811 12:36:52.508116 29115 catalog_manager.cc:4841] TS dde40baade934b509310a4ccc878cc4b (127.3.204.66:41589): tablet 835eb4017bce4986a7d788c9bb99444a (table test-table [id=de9cbc0e02f04900afce62ff3f8b9e12]) successfully deleted
I20260811 12:36:52.508548 29112 catalog_manager.cc:4841] TS 9d9e0e0b9ca242b3b0bea47fc26bda7b (127.3.204.67:45173): tablet 835eb4017bce4986a7d788c9bb99444a (table test-table [id=de9cbc0e02f04900afce62ff3f8b9e12]) successfully deleted
I20260811 12:36:52.602614 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 29232
W20260811 12:36:52.628187 29114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.65:44595 (error 108)
I20260811 12:36:52.628974 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 29508
I20260811 12:36:52.630409 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.629865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52726)
0811 12:36:52.630110 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.630253 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260811 12:36:52.630707 29114 catalog_manager.cc:4584] TS 44d2f7a3681e4e53af87f8426cfd4977 (127.3.204.65:44595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 835eb4017bce4986a7d788c9bb99444a: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
I20260811 12:36:52.645623 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.629738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.3.204.66:41589)
0811 12:36:52.629945 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.629952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.630070 (+ 118us) server_negotiation.cc:407] Connection header received
0811 12:36:52.630320 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.630326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.630412 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.630522 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:52.632069 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.633359 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.634701 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.635417 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.635552 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.638210 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:52.638229 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:52.638252 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:52.638306 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:52.642457 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.643329 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.643338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.643342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.643543 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.643914 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.643919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.643923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.644445 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:52.644796 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.644953 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.645209 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:36:52.646559 29742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:52.684569 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.683911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52738)
0811 12:36:52.684194 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.684357 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260811 12:36:52.685671 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.685293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52752)
0811 12:36:52.685457 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.685543 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 12:36:52.746165 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.745609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52764)
0811 12:36:52.745849 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.745975 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260811 12:36:52.747253 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.746884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52776)
0811 12:36:52.747043 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.747133 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 12:36:52.761405 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 29508
I20260811 12:36:52.787176 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 29753
I20260811 12:36:52.800818 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.787942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.3.204.67:45173)
0811 12:36:52.788155 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 12:36:52.788161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:36:52.788181 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:36:52.788373 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:36:52.788377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:36:52.788445 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:36:52.788529 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:36:52.789886 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.790637 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.791981 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:36:52.792576 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:36:52.792693 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:36:52.795269 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:36:52.795284 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:36:52.795301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:36:52.795337 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:36:52.797974 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.798689 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.798696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.798699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.798828 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:36:52.799214 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:36:52.799218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:36:52.799220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:36:52.799596 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:36:52.799892 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0811 12:36:52.800228 (+ 336us) server_negotiation.cc:299] Negotiation successful
0811 12:36:52.800453 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 12:36:52.801473 29977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:52.828472 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.827874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52788)
0811 12:36:52.828157 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.828268 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260811 12:36:52.829586 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:52.829179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52802)
0811 12:36:52.829390 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:52.829470 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260811 12:36:52.889858 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 29753
I20260811 12:36:52.913125 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 29069
I20260811 12:36:52.914362 29218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:36:53.027814 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:53.027148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:44595 (local address 127.0.0.1:52806)
0811 12:36:53.027488 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:53.027593 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260811 12:36:53.027871 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 29069
I20260811 12:36:53.058143 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46081
--webserver_interface=127.3.204.126
--webserver_port=46727
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:46081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:53.411754 30065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:53.412065 30065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:53.412199 30065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:53.419528 30065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:36:53.419656 30065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:53.419700 30065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:53.419759 30065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:36:53.419800 30065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:36:53.430017 30065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:46081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=46727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:53.431370 30065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:53.432782 30065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:53.444587 30090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:53.444770 30087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:53.444749 30079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:53.445865 30065 server_base.cc:1034] running on GCE node
I20260811 12:36:53.446681 30065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:53.448215 30065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:53.449456 30065 hybrid_clock.cc:648] HybridClock initialized: now 1786451813449397 us; error 47 us; skew 500 ppm
I20260811 12:36:53.449839 30065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:53 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:53.453816 30065 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:36:53.455684 30065 webserver.cc:460] Webserver started at http://127.3.204.126:46727/ using document root <none> and password file <none>
I20260811 12:36:53.456518 30065 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:53.456651 30065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:53.462610 30065 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 12:36:53.465965 30099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:53.467510 30065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:36:53.467689 30065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8"
format_stamp: "Formatted at 2026-08-11 12:36:47 on dist-test-slave-lgr5"
server_key: "12538c978bf6a3fc857344254f6fce75"
server_key_iv: "84a3f6302c8210fcbef72b1369a1215a"
server_key_version: "encryptionkey@0"
I20260811 12:36:53.468779 30065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:53.490000 30065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:53.494542 30065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:53.494823 30065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:53.513396 30065 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:46081
I20260811 12:36:53.513402 30195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:46081 every 8 connection(s)
I20260811 12:36:53.515060 30065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:36:53.517086 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 30065
I20260811 12:36:53.518097 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:44595
--local_ip_for_outbound_sockets=127.3.204.65
--tserver_master_addrs=127.3.204.126:46081
--webserver_port=39215
--webserver_interface=127.3.204.65
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:36:53.524695 30196 sys_catalog.cc:263] Verifying existing consensus state
I20260811 12:36:53.529204 30196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Bootstrap starting.
I20260811 12:36:53.550500 30196 log.cc:824] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Log is configured to *not* fsync() on all Append() calls
I20260811 12:36:53.567795 30196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:36:53.568426 30196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8: Bootstrap complete.
I20260811 12:36:53.577991 30196 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:53.578890 30196 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad60ebf94bc94228a8b1a1c4fd255ea8, State: Initialized, Role: FOLLOWER
I20260811 12:36:53.579541 30196 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:53.579756 30196 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:36:53.579872 30196 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:36:53.580040 30196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 2 FOLLOWER]: Advancing to term 3
I20260811 12:36:53.584179 30196 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:53.584682 30196 leader_election.cc:302] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad60ebf94bc94228a8b1a1c4fd255ea8; no voters:
I20260811 12:36:53.585321 30196 leader_election.cc:288] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 12:36:53.585475 30202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 12:36:53.586603 30202 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [term 3 LEADER]: Becoming Leader. State: Replica: ad60ebf94bc94228a8b1a1c4fd255ea8, State: Running, Role: LEADER
I20260811 12:36:53.587410 30202 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } }
I20260811 12:36:53.587781 30196 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:36:53.590116 30206 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad60ebf94bc94228a8b1a1c4fd255ea8. Latest consensus state: current_term: 3 leader_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } } }
I20260811 12:36:53.590512 30206 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:53.591087 30203 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad60ebf94bc94228a8b1a1c4fd255ea8" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46081 } } }
I20260811 12:36:53.591359 30203 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad60ebf94bc94228a8b1a1c4fd255ea8 [sys.catalog]: This master's current role is: LEADER
I20260811 12:36:53.596287 30215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:36:53.605582 30215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:36:53.607256 30215 catalog_manager.cc:1259] Loaded cluster ID: 0e787c31455c4a4abc11763297f5e54e
I20260811 12:36:53.607412 30215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260811 12:36:53.609840 30215 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff7417a19ad google::LogMessage::Fail() at ??:0
@ 0x7ff7417a488a google::LogMessage::SendToLog() at ??:0
@ 0x7ff7417a14de google::LogMessage::Flush() at ??:0
@ 0x7ff7417a2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260811 12:36:53.624392 30235 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7ff7571d8512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff7571d5976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff75725d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff75725d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff7443ed345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff7443f091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff7443f0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff7443cbeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff7476a56db start_thread at ??:0
@ 0x7ff74063b61f clone at ??:0
W20260811 12:36:53.926016 30199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:53.926317 30199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:53.926415 30199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:53.933871 30199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:53.933995 30199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:53.934119 30199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:36:53.944123 30199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:44595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=39215
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:53.945395 30199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:53.946700 30199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:53.959738 30274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:53.959859 30280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:53.960047 30272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:53.960935 30199 server_base.cc:1034] running on GCE node
I20260811 12:36:53.961568 30199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:53.962567 30199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:53.963771 30199 hybrid_clock.cc:648] HybridClock initialized: now 1786451813963714 us; error 47 us; skew 500 ppm
I20260811 12:36:53.964160 30199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:53 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:53.967324 30199 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:36:53.968977 30199 webserver.cc:460] Webserver started at http://127.3.204.65:39215/ using document root <none> and password file <none>
I20260811 12:36:53.969709 30199 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:53.969836 30199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:53.983289 30199 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.002s sys 0.007s
I20260811 12:36:53.989662 30294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:53.991127 30199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:36:53.991293 30199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "44d2f7a3681e4e53af87f8426cfd4977"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "4aa95f85ce8aa05f7d5785ff5fce5f51"
server_key_iv: "ccd563cd9a264bc8c72ee01fd327c826"
server_key_version: "encryptionkey@0"
I20260811 12:36:53.992391 30199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:54.005839 30199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:54.010008 30199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:54.010288 30199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:54.011358 30199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:54.013291 30302 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260811 12:36:54.018807 30302 ts_tablet_manager.cc:1852] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260811 12:36:54.019838 30302 ts_tablet_manager.cc:1905] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:36:54.027575 30302 ts_tablet_manager.cc:1918] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:36:54.027801 30302 log.cc:1196] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/835eb4017bce4986a7d788c9bb99444a
I20260811 12:36:54.028326 30302 ts_tablet_manager.cc:1939] T 835eb4017bce4986a7d788c9bb99444a P 44d2f7a3681e4e53af87f8426cfd4977: Deleting consensus metadata
I20260811 12:36:54.028760 30199 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260811 12:36:54.028908 30199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260811 12:36:54.029057 30199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:54.029142 30199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:54.064144 30199 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:44595
I20260811 12:36:54.064254 30475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:44595 every 8 connection(s)
I20260811 12:36:54.065575 30199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:36:54.070350 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:54.068536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:35467)
0811 12:36:54.069264 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:54.069519 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":256,"threads_started":1}
W20260811 12:36:54.070968 30476 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:46081 (0 consecutive failures): Network error: Failed to ping master at 127.3.204.126:46081: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
I20260811 12:36:54.071789 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:54.071467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:48899)
0811 12:36:54.071617 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:54.071685 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 12:36:54.072628 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:54.072295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:60085)
0811 12:36:54.072430 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:54.072510 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 12:36:54.072839 30476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 12:36:54.073525 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 30199
I20260811 12:36:54.074107 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:41589
--local_ip_for_outbound_sockets=127.3.204.66
--tserver_master_addrs=127.3.204.126:46081
--webserver_port=40135
--webserver_interface=127.3.204.66
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:36:54.423879 30484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:54.424255 30484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:54.424412 30484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:54.431773 30484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:54.431936 30484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:54.432121 30484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:36:54.442507 30484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:41589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=40135
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:54.443892 30484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:54.445283 30484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:54.458426 30491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:54.458393 30492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:54.458359 30494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:54.459188 30484 server_base.cc:1034] running on GCE node
I20260811 12:36:54.459796 30484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:54.460837 30484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:54.462039 30484 hybrid_clock.cc:648] HybridClock initialized: now 1786451814461991 us; error 38 us; skew 500 ppm
I20260811 12:36:54.462361 30484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:54 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:54.465660 30484 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:36:54.467226 30484 webserver.cc:460] Webserver started at http://127.3.204.66:40135/ using document root <none> and password file <none>
I20260811 12:36:54.467934 30484 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:54.468045 30484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:54.473588 30484 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:36:54.477130 30508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:54.478500 30484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:36:54.478670 30484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "dde40baade934b509310a4ccc878cc4b"
format_stamp: "Formatted at 2026-08-11 12:36:48 on dist-test-slave-lgr5"
server_key: "d238b65a13f82c692a35b9f279ebaab1"
server_key_iv: "71440b8e7fdc118a7dfe796eef125b4b"
server_key_version: "encryptionkey@0"
I20260811 12:36:54.479671 30484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:54.512315 30484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:54.516546 30484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:54.516827 30484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:54.517885 30484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:54.519275 30484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:54.519377 30484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:54.519500 30484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:54.519577 30484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:54.551060 30484 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:41589
I20260811 12:36:54.551119 30716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:41589 every 8 connection(s)
I20260811 12:36:54.552558 30484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:36:54.553294 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 30484
I20260811 12:36:54.554061 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:45173
--local_ip_for_outbound_sockets=127.3.204.67
--tserver_master_addrs=127.3.204.126:46081
--webserver_port=39313
--webserver_interface=127.3.204.67
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:36:54.557974 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:54.556013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:44067)
0811 12:36:54.556836 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:54.557100 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":325,"threads_started":1}
W20260811 12:36:54.558638 30717 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:46081 (0 consecutive failures): Network error: Failed to ping master at 127.3.204.126:46081: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
I20260811 12:36:54.559495 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:54.559163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:33709)
0811 12:36:54.559313 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:54.559384 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 12:36:54.560335 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:54.559997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:54117)
0811 12:36:54.560153 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:54.560232 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 12:36:54.560580 30717 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 12:36:54.892343 30726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:36:54.892632 30726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:36:54.892725 30726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:36:54.900096 30726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:36:54.900207 30726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:36:54.900324 30726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:36:54.910172 30726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:45173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=39313
--tserver_master_addrs=127.3.204.126:46081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:36:54.911582 30726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:36:54.912961 30726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:36:54.926198 30734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:54.926250 30738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:36:54.926204 30735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:36:54.926995 30726 server_base.cc:1034] running on GCE node
I20260811 12:36:54.927623 30726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:36:54.928673 30726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:36:54.929876 30726 hybrid_clock.cc:648] HybridClock initialized: now 1786451814929822 us; error 50 us; skew 500 ppm
I20260811 12:36:54.930205 30726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:36:54 dist-test-slave-lgr5 krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:36:54.933462 30726 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:36:54.935019 30726 webserver.cc:460] Webserver started at http://127.3.204.67:39313/ using document root <none> and password file <none>
I20260811 12:36:54.935701 30726 fs_manager.cc:359] Metadata directory not provided
I20260811 12:36:54.935806 30726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:36:54.941397 30726 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:36:54.944162 30750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:54.945555 30726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:36:54.945703 30726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "9d9e0e0b9ca242b3b0bea47fc26bda7b"
format_stamp: "Formatted at 2026-08-11 12:36:49 on dist-test-slave-lgr5"
server_key: "f987af3b6391c6872a89a5ce2f54d48d"
server_key_iv: "a090f6157c622cc3f51db87ccc3239e3"
server_key_version: "encryptionkey@0"
I20260811 12:36:54.946679 30726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:36:54.975111 30726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:36:54.979461 30726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:36:54.979705 30726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:36:54.980693 30726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:36:54.982074 30726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:36:54.982158 30726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:54.982237 30726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:36:54.982290 30726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:36:55.014339 30726 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:45173
I20260811 12:36:55.014421 30947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:45173 every 8 connection(s)
I20260811 12:36:55.015803 30726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:36:55.020617 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:55.018648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:59823)
0811 12:36:55.019400 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:55.019644 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":311,"threads_started":1}
W20260811 12:36:55.021337 30948 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:46081 (0 consecutive failures): Network error: Failed to ping master at 127.3.204.126:46081: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
I20260811 12:36:55.022130 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:55.021818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:40829)
0811 12:36:55.021948 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:55.022021 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 12:36:55.022955 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:55.022599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:35785)
0811 12:36:55.022724 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:55.022840 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 12:36:55.023200 30948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 12:36:55.024116 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 30726
I20260811 12:36:55.074241 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:55.073459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:59603)
0811 12:36:55.073926 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:55.074020 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":219,"threads_started":1}
I20260811 12:36:55.561930 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:55.561137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:38227)
0811 12:36:55.561603 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:55.561699 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":217,"threads_started":1}
I20260811 12:36:56.024590 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:56.023748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:58471)
0811 12:36:56.024230 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:56.024326 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":217,"threads_started":1}
I20260811 12:36:56.075843 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:56.075124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:57457)
0811 12:36:56.075592 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:56.075682 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":240,"threads_started":1}
I20260811 12:36:56.563522 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:56.562812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:37539)
0811 12:36:56.563244 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:56.563330 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":210,"threads_started":1}
I20260811 12:36:57.026302 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:57.025527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:58599)
0811 12:36:57.026032 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:57.026126 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":241,"threads_started":1}
I20260811 12:36:57.077612 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:57.076839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:38749)
0811 12:36:57.077314 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:57.077404 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":229,"threads_started":1}
I20260811 12:36:57.565230 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:57.564439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:60301)
0811 12:36:57.564944 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:57.565048 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":260,"threads_started":1}
I20260811 12:36:58.028030 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:58.027259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:56375)
0811 12:36:58.027728 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:58.027819 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":253,"threads_started":1}
I20260811 12:36:58.079361 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:58.078547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:53367)
0811 12:36:58.079073 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:58.079186 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":239,"threads_started":1}
I20260811 12:36:58.566977 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:58.566211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:46307)
0811 12:36:58.566710 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:58.566800 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":267,"threads_started":1}
I20260811 12:36:59.029940 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:59.029028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:38253)
0811 12:36:59.029597 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:59.029697 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":302,"threads_started":1}
I20260811 12:36:59.081048 30969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:59.080307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:48737)
0811 12:36:59.080770 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:59.080865 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":230,"threads_started":1}
I20260811 12:36:59.568816 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:36:59.567997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:38255)
0811 12:36:59.568537 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0811 12:36:59.568635 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":286,"threads_started":1}
I20260811 12:37:00.031930 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:00.030946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:48985)
0811 12:37:00.031536 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:00.031686 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":250,"threads_started":1}
I20260811 12:37:00.082770 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:00.082023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:34783)
0811 12:37:00.082519 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:00.082605 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":257,"threads_started":1}
I20260811 12:37:00.570415 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:00.569669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:58599)
0811 12:37:00.570148 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:00.570245 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260811 12:37:01.033739 30976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:01.032916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:49279)
0811 12:37:01.033464 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:01.033570 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":301,"threads_started":1}
I20260811 12:37:01.084673 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:01.083733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:33143)
0811 12:37:01.084317 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:01.084402 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":252,"threads_started":1}
I20260811 12:37:01.572122 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:01.571315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:37605)
0811 12:37:01.571798 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:01.571886 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":219,"threads_started":1}
I20260811 12:37:02.035692 30979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:02.034819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:55843)
0811 12:37:02.035382 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:02.035475 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":267,"threads_started":1}
I20260811 12:37:02.086463 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:02.085690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:39997)
0811 12:37:02.086196 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:02.086284 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":287,"threads_started":1}
I20260811 12:37:02.573817 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:02.573015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:38659)
0811 12:37:02.573553 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:02.573644 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":310,"threads_started":1}
I20260811 12:37:03.037520 30984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:03.036707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:47015)
0811 12:37:03.037216 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:03.037318 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":254,"threads_started":1}
I20260811 12:37:03.088174 30988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:03.087315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:57885)
0811 12:37:03.087844 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:03.087948 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
I20260811 12:37:03.575588 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:03.574749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:39255)
0811 12:37:03.575288 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:03.575404 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260811 12:37:04.039343 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:04.038467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:53187)
0811 12:37:04.039042 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:04.039139 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":279,"threads_started":1}
I20260811 12:37:04.089931 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:04.089190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:43645)
0811 12:37:04.089661 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:04.089746 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260811 12:37:04.577375 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:04.576649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:41915)
0811 12:37:04.577118 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:04.577203 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
I20260811 12:37:05.041270 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:05.040404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:41265)
0811 12:37:05.041010 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:05.041102 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":354,"threads_started":1}
I20260811 12:37:05.091837 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:05.091000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:39905)
0811 12:37:05.091543 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:05.091636 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":283,"threads_started":1}
I20260811 12:37:05.579190 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:05.578392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:35693)
0811 12:37:05.578874 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:05.578981 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
I20260811 12:37:06.043090 30999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:06.042262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:57423)
0811 12:37:06.042795 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:06.042909 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":293,"threads_started":1}
I20260811 12:37:06.093504 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:06.092776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:53033)
0811 12:37:06.093238 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:06.093334 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260811 12:37:06.581059 31002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:06.580253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:45859)
0811 12:37:06.580805 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:06.580895 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260811 12:37:07.044826 31003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:07.044070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:53789)
0811 12:37:07.044573 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:07.044659 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260811 12:37:07.095371 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:07.094548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:46265)
0811 12:37:07.095061 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:07.095173 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":238,"threads_started":1}
I20260811 12:37:07.583003 31007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:07.582076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:48765)
0811 12:37:07.582650 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:07.582784 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":291,"threads_started":1}
I20260811 12:37:08.046737 31008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:08.045887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:53931)
0811 12:37:08.046414 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:08.046520 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":260,"threads_started":1}
I20260811 12:37:08.097239 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:08.096423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:35791)
0811 12:37:08.096917 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:08.097023 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":233,"threads_started":1}
I20260811 12:37:08.584745 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:08.583931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:39967)
0811 12:37:08.584447 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:08.584548 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":259,"threads_started":1}
I20260811 12:37:09.048537 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:09.047716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:44219)
0811 12:37:09.048223 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:09.048316 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":233,"threads_started":1}
I20260811 12:37:09.099318 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:09.098488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.65:54795)
0811 12:37:09.099021 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:09.099126 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":262,"threads_started":1}
I20260811 12:37:09.586732 31022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:09.585922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.66:55265)
0811 12:37:09.586432 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:09.586539 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260811 12:37:10.027446 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 30199
I20260811 12:37:10.050594 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:10.049747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46081 (local address 127.3.204.67:48277)
0811 12:37:10.050214 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:10.050353 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":249,"threads_started":1}
I20260811 12:37:10.051918 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:10.028742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.3.204.65:44595)
0811 12:37:10.029341 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:10.029356 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:10.029626 (+ 270us) server_negotiation.cc:407] Connection header received
0811 12:37:10.031425 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:10.031436 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:10.031873 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:10.032237 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:10.033713 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:10.035042 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:10.036546 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:10.037266 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:10.037425 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:10.045157 (+ 7732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:10.045166 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:10.045183 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:10.045236 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:10.048149 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:10.048922 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:10.048930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:10.048937 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:10.049107 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:10.049510 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:10.049514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:10.049517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:10.050314 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:10.050598 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:10.051171 (+ 573us) server_negotiation.cc:299] Negotiation successful
0811 12:37:10.051633 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":242,"threads_started":1}
I20260811 12:37:10.053396 30467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:10.139273 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 30199
I20260811 12:37:10.164386 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 30484
I20260811 12:37:10.182623 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:10.165448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.3.204.66:41589)
0811 12:37:10.165950 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:10.165966 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:10.166006 (+ 40us) server_negotiation.cc:407] Connection header received
0811 12:37:10.167503 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:10.167513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:10.167952 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:10.168304 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:10.169782 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:10.171064 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:10.172583 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:10.173267 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:10.173398 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:10.176025 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:10.176033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:10.176051 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:10.176136 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:10.179302 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:10.179983 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:10.179989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:10.179997 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:10.180232 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:10.180561 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:10.180564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:10.180566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:10.181157 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:10.181402 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:10.181850 (+ 448us) server_negotiation.cc:299] Negotiation successful
0811 12:37:10.182279 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260811 12:37:10.184283 30706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:10.268720 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 30484
I20260811 12:37:10.297668 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 30726
I20260811 12:37:10.315510 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:10.298674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.3.204.67:45173)
0811 12:37:10.299247 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:10.299262 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:10.299305 (+ 43us) server_negotiation.cc:407] Connection header received
0811 12:37:10.301059 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:10.301071 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:10.301519 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:10.301824 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:10.303279 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:10.304639 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:10.305903 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:10.306585 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:10.306722 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:10.309363 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:10.309372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:10.309390 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:10.309432 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:10.312483 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:10.313196 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:10.313203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:10.313211 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:10.313393 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:10.313694 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:10.313697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:10.313699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:10.314252 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:10.314479 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:10.314821 (+ 342us) server_negotiation.cc:299] Negotiation successful
0811 12:37:10.315205 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":258,"threads_started":1}
I20260811 12:37:10.317022 30942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:10.398885 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 30726
2026-08-11T12:37:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24797 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:37:10 dist-test-slave-lgr5 krb5kdc[31049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:37:10 dist-test-slave-lgr5 krb5kdc[31049](info): set up 2 sockets
Aug 11 12:37:10 dist-test-slave-lgr5 krb5kdc[31049](info): commencing operation
krb5kdc: starting...
W20260811 12:37:11.685698 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:37:11 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:37:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:37:11.982105 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33833
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:37:12.330927 31069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:12.331252 31069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:12.331372 31069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:12.338886 31069 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 12:37:12.339015 31069 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 12:37:12.339052 31069 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 12:37:12.339099 31069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:37:12.339156 31069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:12.339267 31069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:12.339318 31069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:37:12.339386 31069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:37:12.349820 31069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:33833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:12.351181 31069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:12.352626 31069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:12.365116 31077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:12.365116 31069 server_base.cc:1034] running on GCE node
W20260811 12:37:12.365108 31080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:12.365165 31083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:12.366315 31069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:12.367733 31069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:12.368996 31069 hybrid_clock.cc:648] HybridClock initialized: now 1786451832368925 us; error 33 us; skew 500 ppm
I20260811 12:37:12.369349 31069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:12 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451832, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:12.372483 31069 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:37:12.373927 31069 webserver.cc:460] Webserver started at http://127.3.204.126:33961/ using document root <none> and password file <none>
I20260811 12:37:12.374627 31069 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:12.374735 31069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:12.375068 31069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:37:12.378110 31069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "0a8b307ea77640da90db2b1095ee6275"
format_stamp: "Formatted at 2026-08-11 12:37:12 on dist-test-slave-lgr5"
server_key: "e9400dcc57d2a2e7d773ee42306086f2"
server_key_iv: "704d6f401dc82fa995d5d18f4bf3b8f8"
server_key_version: "encryptionkey@0"
I20260811 12:37:12.378937 31069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "0a8b307ea77640da90db2b1095ee6275"
format_stamp: "Formatted at 2026-08-11 12:37:12 on dist-test-slave-lgr5"
server_key: "e9400dcc57d2a2e7d773ee42306086f2"
server_key_iv: "704d6f401dc82fa995d5d18f4bf3b8f8"
server_key_version: "encryptionkey@0"
I20260811 12:37:12.384608 31069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:37:12.388170 31097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:12.389577 31069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:37:12.389744 31069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "0a8b307ea77640da90db2b1095ee6275"
format_stamp: "Formatted at 2026-08-11 12:37:12 on dist-test-slave-lgr5"
server_key: "e9400dcc57d2a2e7d773ee42306086f2"
server_key_iv: "704d6f401dc82fa995d5d18f4bf3b8f8"
server_key_version: "encryptionkey@0"
I20260811 12:37:12.389974 31069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:12.402719 31069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:12.406723 31069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:12.407027 31069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:12.424432 31069 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:33833
I20260811 12:37:12.424430 31184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:33833 every 8 connection(s)
I20260811 12:37:12.425891 31069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:37:12.429773 31185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:37:12.431878 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 31069
I20260811 12:37:12.432099 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:37:12.432783 3889 external_mini_cluster.cc:1428] Setting key c36a27e67df888cdfd59c4681a4aacd8
I20260811 12:37:12.441651 31185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Bootstrap starting.
I20260811 12:37:12.445580 31185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Neither blocks nor log segments found. Creating new log.
I20260811 12:37:12.446942 31185 log.cc:824] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Log is configured to *not* fsync() on all Append() calls
Aug 11 12:37:12 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:37:12.450322 31185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: No bootstrap required, opened a new log
I20260811 12:37:12.458277 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:12.434725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.3.204.126:33833)
0811 12:37:12.435484 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:12.435496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:12.435538 (+ 42us) server_negotiation.cc:407] Connection header received
0811 12:37:12.437280 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:12.437304 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:12.437686 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:12.438044 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:12.439961 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:12.441352 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:12.443232 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:12.443939 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:12.444070 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:12.449303 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:12.449315 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:12.449337 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:12.449398 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:12.453339 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:12.454145 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:12.454154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:12.454163 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:12.454413 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:12.454759 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:12.454764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:12.454768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:12.455562 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:12.455881 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:12.456336 (+ 455us) server_negotiation.cc:299] Negotiation successful
0811 12:37:12.456837 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":352,"threads_started":1}
I20260811 12:37:12.460712 31185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:12.461202 31185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:37:12.461311 31185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8b307ea77640da90db2b1095ee6275, State: Initialized, Role: FOLLOWER
I20260811 12:37:12.461905 31185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:12.462116 31185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:37:12.462250 31185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:37:12.462392 31185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:37:12.464972 31185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:12.465457 31185 leader_election.cc:302] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a8b307ea77640da90db2b1095ee6275; no voters:
I20260811 12:37:12.466038 31185 leader_election.cc:288] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:37:12.466159 31192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:37:12.466846 31192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 1 LEADER]: Becoming Leader. State: Replica: 0a8b307ea77640da90db2b1095ee6275, State: Running, Role: LEADER
I20260811 12:37:12.467381 31192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:12.468181 31185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:37:12.470505 31194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8b307ea77640da90db2b1095ee6275. Latest consensus state: current_term: 1 leader_uuid: "0a8b307ea77640da90db2b1095ee6275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } } }
I20260811 12:37:12.470515 31193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a8b307ea77640da90db2b1095ee6275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } } }
I20260811 12:37:12.471073 31193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: This master's current role is: LEADER
I20260811 12:37:12.471681 31194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: This master's current role is: LEADER
I20260811 12:37:12.471887 31209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:37:12.476964 31209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:37:12.486451 31209 catalog_manager.cc:1347] Generated new cluster ID: 988595181862414597631268084daf5f
I20260811 12:37:12.486560 31209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:37:12.530880 31209 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:37:12.532804 31209 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:37:12.543802 31209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Generated new TSK 0
I20260811 12:37:12.544696 31209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:37:12.625705 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33833
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:37:12.968331 31226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:12.968631 31226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:12.968768 31226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:12.976480 31226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:12.976600 31226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:12.976758 31226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:37:12.986672 31226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:12.988008 31226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:12.989388 31226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:13.003086 31233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:13.003146 31234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:13.003134 31238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:13.003925 31226 server_base.cc:1034] running on GCE node
I20260811 12:37:13.004645 31226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:13.005698 31226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:13.006940 31226 hybrid_clock.cc:648] HybridClock initialized: now 1786451833006882 us; error 45 us; skew 500 ppm
I20260811 12:37:13.007299 31226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:13 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:13.010572 31226 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:37:13.012179 31226 webserver.cc:460] Webserver started at http://127.3.204.65:46639/ using document root <none> and password file <none>
I20260811 12:37:13.012907 31226 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:13.013031 31226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:13.013351 31226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:37:13.016624 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "af5fe9d1f56244f990d8a5e608b13fdf"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "be2d5fa4ccdc6d947021af99aabea016"
server_key_iv: "311f9914176b63f025eabb9a7e4ddb99"
server_key_version: "encryptionkey@0"
I20260811 12:37:13.017416 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "af5fe9d1f56244f990d8a5e608b13fdf"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "be2d5fa4ccdc6d947021af99aabea016"
server_key_iv: "311f9914176b63f025eabb9a7e4ddb99"
server_key_version: "encryptionkey@0"
I20260811 12:37:13.022809 31226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:37:13.026435 31252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:13.027738 31226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:37:13.027900 31226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "af5fe9d1f56244f990d8a5e608b13fdf"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "be2d5fa4ccdc6d947021af99aabea016"
server_key_iv: "311f9914176b63f025eabb9a7e4ddb99"
server_key_version: "encryptionkey@0"
I20260811 12:37:13.028124 31226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:13.040874 31226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:13.045418 31226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:13.045724 31226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:13.046777 31226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:37:13.048308 31226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:37:13.048405 31226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:13.048534 31226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:37:13.048588 31226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:13.080628 31226 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:46501
I20260811 12:37:13.080682 31466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:46501 every 8 connection(s)
I20260811 12:37:13.081986 31226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:37:13.087242 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 31226
I20260811 12:37:13.087476 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:37:13.088310 3889 external_mini_cluster.cc:1428] Setting key 9407758ee6f647be5a0b85b380948a3c
Aug 11 12:37:13 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:37:13.105321 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:13.084706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:35531 (local address 127.3.204.126:33833)
0811 12:37:13.085171 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:13.085176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:13.086060 (+ 884us) server_negotiation.cc:407] Connection header received
0811 12:37:13.088545 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:13.088550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:13.088634 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:13.088735 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:13.090625 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:13.091395 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:13.093699 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:13.094160 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:13.094293 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:13.098626 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:13.098634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:13.098639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:13.098684 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:13.101863 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:13.102780 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:13.102785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:13.102788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:13.102942 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:13.103420 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:13.103425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:13.103427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:13.103715 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:13.103860 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:13.104695 (+ 835us) server_negotiation.cc:299] Negotiation successful
0811 12:37:13.104940 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":230,"threads_started":1}
I20260811 12:37:13.106331 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:13.085167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:35531)
0811 12:37:13.085890 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:13.085938 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:13.088157 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:13.089026 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:13.089035 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:13.089703 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:37:13.090418 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:13.090447 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:13.091569 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:13.091574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:13.093534 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:13.093559 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:13.094405 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:13.094412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:13.094694 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:13.095497 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:37:13.095529 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:37:13.098320 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:37:13.102071 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:13.102081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:13.102098 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:13.102626 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:13.103049 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:13.103052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:13.103055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:13.103268 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:13.103868 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:37:13.103876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:37:13.104289 (+ 413us) client_negotiation.cc:769] Sending connection context
0811 12:37:13.104550 (+ 261us) client_negotiation.cc:240] Negotiation successful
0811 12:37:13.105153 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":282,"threads_started":1}
I20260811 12:37:13.108068 31467 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33833
I20260811 12:37:13.108462 31467 heartbeater.cc:464] Registering TS with master...
I20260811 12:37:13.109452 31467 heartbeater.cc:511] Master 127.3.204.126:33833 requested a full tablet report, sending...
I20260811 12:37:13.112202 31124 ts_manager.cc:194] Registered new tserver with Master: af5fe9d1f56244f990d8a5e608b13fdf (127.3.204.65:46501)
I20260811 12:37:13.114169 31124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:35531
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:37:13.159734 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33833
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:37:13.502621 31480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:13.502916 31480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:13.503036 31480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:13.510365 31480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:13.510484 31480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:13.510627 31480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:37:13.520617 31480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:13.522010 31480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:13.523356 31480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:13.537737 31490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:13.537704 31492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:13.537667 31495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:13.538640 31480 server_base.cc:1034] running on GCE node
I20260811 12:37:13.539268 31480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:13.540303 31480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:13.541497 31480 hybrid_clock.cc:648] HybridClock initialized: now 1786451833541448 us; error 39 us; skew 500 ppm
I20260811 12:37:13.541812 31480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:13 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:13.544970 31480 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:37:13.546540 31480 webserver.cc:460] Webserver started at http://127.3.204.66:44809/ using document root <none> and password file <none>
I20260811 12:37:13.547253 31480 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:13.547375 31480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:13.547690 31480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:37:13.550693 31480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "d6b8bb3c552f4efa82899e5ef437a2b0"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "dd7960f815a9b1786c4d11fe20c84bb4"
server_key_iv: "09b5c73d4fb89773416b5e92b1594f95"
server_key_version: "encryptionkey@0"
I20260811 12:37:13.551461 31480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "d6b8bb3c552f4efa82899e5ef437a2b0"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "dd7960f815a9b1786c4d11fe20c84bb4"
server_key_iv: "09b5c73d4fb89773416b5e92b1594f95"
server_key_version: "encryptionkey@0"
I20260811 12:37:13.556838 31480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260811 12:37:13.560617 31508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:13.562222 31480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:37:13.562404 31480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "d6b8bb3c552f4efa82899e5ef437a2b0"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "dd7960f815a9b1786c4d11fe20c84bb4"
server_key_iv: "09b5c73d4fb89773416b5e92b1594f95"
server_key_version: "encryptionkey@0"
I20260811 12:37:13.562601 31480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:13.585850 31480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:13.589890 31480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:13.590162 31480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:13.591149 31480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:37:13.592658 31480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:37:13.592744 31480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:13.592864 31480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:37:13.592918 31480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:13.623746 31480 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:42875
I20260811 12:37:13.623791 31716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:42875 every 8 connection(s)
I20260811 12:37:13.625077 31480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:37:13.630038 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 31480
I20260811 12:37:13.630218 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:37:13.630650 3889 external_mini_cluster.cc:1428] Setting key f7534ad23f839b5246673bd40ae2619e
Aug 11 12:37:13 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:37:13.644646 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:13.627609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:37693 (local address 127.3.204.126:33833)
0811 12:37:13.628054 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:13.628059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:13.629085 (+ 1026us) server_negotiation.cc:407] Connection header received
0811 12:37:13.630978 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:13.630984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:13.631067 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:13.631160 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:13.632911 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:13.633623 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:13.635138 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:13.635550 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:13.635655 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:13.639117 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:13.639123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:13.639126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:13.639158 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:13.641608 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:13.642323 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:13.642327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:13.642329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:13.642462 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:13.642874 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:13.642877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:13.642879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:13.643147 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:13.643280 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:13.644011 (+ 731us) server_negotiation.cc:299] Negotiation successful
0811 12:37:13.644270 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":214,"threads_started":1}
I20260811 12:37:13.645635 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:13.628217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:37693)
0811 12:37:13.628922 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:13.628973 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:13.630682 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:13.631433 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:13.631445 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:13.632052 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:37:13.632731 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:13.632749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:13.633763 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:13.633767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:13.635000 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:13.635013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:13.635759 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:13.635767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:13.635986 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:13.636655 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:37:13.636678 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:37:13.638897 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:37:13.641746 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:13.641755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:13.641774 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:13.642197 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:13.642567 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:13.642572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:13.642575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:13.642759 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:13.643272 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:37:13.643280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:37:13.643630 (+ 350us) client_negotiation.cc:769] Sending connection context
0811 12:37:13.643872 (+ 242us) client_negotiation.cc:240] Negotiation successful
0811 12:37:13.644448 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":263,"threads_started":1}
I20260811 12:37:13.647123 31719 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33833
I20260811 12:37:13.647445 31719 heartbeater.cc:464] Registering TS with master...
I20260811 12:37:13.648240 31719 heartbeater.cc:511] Master 127.3.204.126:33833 requested a full tablet report, sending...
I20260811 12:37:13.649874 31124 ts_manager.cc:194] Registered new tserver with Master: d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
I20260811 12:37:13.650969 31124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:37693
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:37:13.696413 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33833
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:37:14.039927 31726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:14.040266 31726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:14.040382 31726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:14.047924 31726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:14.048056 31726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:14.048209 31726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:37:14.058490 31726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:14.059849 31726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:14.061185 31726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:14.074380 31735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:14.074599 31737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:14.074481 31734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:14.075182 31726 server_base.cc:1034] running on GCE node
I20260811 12:37:14.075839 31726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:14.076932 31726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:14.078147 31726 hybrid_clock.cc:648] HybridClock initialized: now 1786451834078090 us; error 51 us; skew 500 ppm
I20260811 12:37:14.078519 31726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:14 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451834, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:14.082149 31726 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:37:14.083698 31726 webserver.cc:460] Webserver started at http://127.3.204.67:41053/ using document root <none> and password file <none>
I20260811 12:37:14.084506 31726 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:14.084621 31726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:14.084940 31726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:37:14.087919 31726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "77c04c7984444e42a4db7a19ab042987"
format_stamp: "Formatted at 2026-08-11 12:37:14 on dist-test-slave-lgr5"
server_key: "99e24c6be1313120843dce135a5b2699"
server_key_iv: "0135bcdc61fd18efc3be29f06b9a222f"
server_key_version: "encryptionkey@0"
I20260811 12:37:14.088770 31726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "77c04c7984444e42a4db7a19ab042987"
format_stamp: "Formatted at 2026-08-11 12:37:14 on dist-test-slave-lgr5"
server_key: "99e24c6be1313120843dce135a5b2699"
server_key_iv: "0135bcdc61fd18efc3be29f06b9a222f"
server_key_version: "encryptionkey@0"
I20260811 12:37:14.094429 31726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:37:14.098346 31748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:14.099812 31726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:37:14.100003 31726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "77c04c7984444e42a4db7a19ab042987"
format_stamp: "Formatted at 2026-08-11 12:37:14 on dist-test-slave-lgr5"
server_key: "99e24c6be1313120843dce135a5b2699"
server_key_iv: "0135bcdc61fd18efc3be29f06b9a222f"
server_key_version: "encryptionkey@0"
I20260811 12:37:14.100275 31726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:14.114893 31726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:14.118163 31467 heartbeater.cc:503] Master 127.3.204.126:33833 was elected leader, sending a full tablet report...
I20260811 12:37:14.119573 31726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:14.119858 31726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:14.120893 31726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:37:14.122339 31726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:37:14.122426 31726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:14.122552 31726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:37:14.122607 31726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:14.154635 31726 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:43673
I20260811 12:37:14.154672 31942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:43673 every 8 connection(s)
I20260811 12:37:14.156038 31726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:37:14.166268 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 31726
I20260811 12:37:14.166540 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:37:14.167037 3889 external_mini_cluster.cc:1428] Setting key b3c86641cb1b1b0aae17e43970710cb3
Aug 11 12:37:14 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451834, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:37:14.176978 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.158741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:56543 (local address 127.3.204.126:33833)
0811 12:37:14.159224 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.159229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.160131 (+ 902us) server_negotiation.cc:407] Connection header received
0811 12:37:14.162020 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.162026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.162125 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.162225 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:14.163951 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.164798 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.166721 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.167158 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.167272 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.171384 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:14.171390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:14.171393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:14.171430 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:14.174201 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.174913 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.174917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.174919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.175051 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.175418 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.175420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.175422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.175695 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:14.175807 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.176515 (+ 708us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.176691 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":239,"threads_started":1}
I20260811 12:37:14.177995 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.159188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:56543)
0811 12:37:14.159928 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:14.159984 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:14.161723 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:14.162497 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:14.162509 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:14.163174 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:37:14.163773 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.163791 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.164960 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.164964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.166577 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.166595 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.167416 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.167424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.167697 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.168444 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:37:14.168480 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:37:14.171167 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:37:14.174346 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:14.174354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:14.174370 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:14.174823 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:14.175128 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:14.175131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:14.175134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:14.175323 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:14.175788 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:37:14.175794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:37:14.176167 (+ 373us) client_negotiation.cc:769] Sending connection context
0811 12:37:14.176412 (+ 245us) client_negotiation.cc:240] Negotiation successful
0811 12:37:14.176834 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":287,"threads_started":1}
I20260811 12:37:14.179489 31943 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33833
I20260811 12:37:14.179800 31943 heartbeater.cc:464] Registering TS with master...
I20260811 12:37:14.180634 31943 heartbeater.cc:511] Master 127.3.204.126:33833 requested a full tablet report, sending...
I20260811 12:37:14.182288 31124 ts_manager.cc:194] Registered new tserver with Master: 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673)
I20260811 12:37:14.182911 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:37:14.183177 31124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:56543
I20260811 12:37:14.201706 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.187840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.3.204.126:33833)
0811 12:37:14.188021 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.188026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.188505 (+ 479us) server_negotiation.cc:407] Connection header received
0811 12:37:14.188651 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.188655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.188725 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.188814 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:14.190318 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.191068 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.192757 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.193254 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.193361 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.196007 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:14.196013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:14.196017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:14.196055 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:14.198772 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.199576 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.199582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.199587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.199733 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.200133 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.200139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.200143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.200536 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:14.200784 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.201146 (+ 362us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.201389 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 12:37:14.206116 31124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:37:14.213238 31124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:37:14.249799 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.226596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.3.204.66:42875)
0811 12:37:14.237033 (+ 10437us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.237046 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.239689 (+ 2643us) server_negotiation.cc:407] Connection header received
0811 12:37:14.239824 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.239834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.240164 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.240366 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:14.242098 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.243680 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.246654 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.248248 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.248380 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.248609 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.249250 (+ 641us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.249446 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10212,"thread_start_us":10063,"threads_started":1}
I20260811 12:37:14.250103 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.226577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:42875 (local address 127.0.0.1:43362)
0811 12:37:14.239179 (+ 12602us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:14.239246 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:14.239437 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:14.241358 (+ 1921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:14.241367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:14.241403 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:14.241874 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.241888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.243875 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.243881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.246451 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.246469 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.248766 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.248773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.248986 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.249044 (+ 58us) client_negotiation.cc:769] Sending connection context
0811 12:37:14.249192 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 12:37:14.249433 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12305,"mutex_wait_us":4,"thread_start_us":14018,"threads_started":1}
I20260811 12:37:14.254560 31605 tablet_service.cc:1467] Processing CreateTablet for tablet 076bbd00fc2a42e3a5b9824c91d05620 (DEFAULT_TABLE table=test-table [id=3139c55cb30f43368a2728819031f4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:37:14.255573 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.227249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:49472)
0811 12:37:14.247839 (+ 20590us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:14.247855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:14.248171 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:14.249055 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:14.249060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:14.249084 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:14.249508 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.249522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.251442 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.251448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.252988 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.253002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.254751 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.254757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.254923 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.254956 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 12:37:14.255066 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 12:37:14.255211 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20473,"thread_start_us":20324,"threads_started":1}
I20260811 12:37:14.256213 31969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.226579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43673 (local address 127.0.0.1:34022)
0811 12:37:14.246020 (+ 19441us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:14.246041 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:14.246155 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:14.248539 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:14.248544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:14.248572 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:14.249046 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.249059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.250941 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.250947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.253009 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.253026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.255101 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.255108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.255284 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.255322 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 12:37:14.255428 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 12:37:14.255573 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19311,"mutex_wait_us":202,"thread_start_us":19426,"threads_started":1}
I20260811 12:37:14.256301 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.227105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.3.204.67:43673)
0811 12:37:14.247495 (+ 20390us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.247503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.247538 (+ 35us) server_negotiation.cc:407] Connection header received
0811 12:37:14.247640 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.247648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.247958 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.248247 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:14.249325 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.250807 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.253202 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.254586 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.254710 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.254935 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.255705 (+ 770us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.255905 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20221,"thread_start_us":20057,"threads_started":1}
I20260811 12:37:14.256458 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.227863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.3.204.65:46501)
0811 12:37:14.246974 (+ 19111us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.246984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.248044 (+ 1060us) server_negotiation.cc:407] Connection header received
0811 12:37:14.248370 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.248377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.248619 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.248798 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:14.249705 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.251303 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.253141 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.254215 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.254344 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.254602 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.255938 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.256134 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18920,"thread_start_us":18768,"threads_started":1}
I20260811 12:37:14.257215 31605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076bbd00fc2a42e3a5b9824c91d05620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:37:14.259613 31344 tablet_service.cc:1467] Processing CreateTablet for tablet 076bbd00fc2a42e3a5b9824c91d05620 (DEFAULT_TABLE table=test-table [id=3139c55cb30f43368a2728819031f4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:37:14.260224 31835 tablet_service.cc:1467] Processing CreateTablet for tablet 076bbd00fc2a42e3a5b9824c91d05620 (DEFAULT_TABLE table=test-table [id=3139c55cb30f43368a2728819031f4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:37:14.260857 31344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076bbd00fc2a42e3a5b9824c91d05620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:37:14.261641 31835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076bbd00fc2a42e3a5b9824c91d05620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:37:14.272742 31974 tablet_bootstrap.cc:492] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Bootstrap starting.
I20260811 12:37:14.274885 31977 tablet_bootstrap.cc:492] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Bootstrap starting.
I20260811 12:37:14.276525 31978 tablet_bootstrap.cc:492] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Bootstrap starting.
I20260811 12:37:14.276849 31974 tablet_bootstrap.cc:654] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Neither blocks nor log segments found. Creating new log.
I20260811 12:37:14.278262 31974 log.cc:824] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:14.278434 31977 tablet_bootstrap.cc:654] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Neither blocks nor log segments found. Creating new log.
I20260811 12:37:14.279477 31978 tablet_bootstrap.cc:654] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Neither blocks nor log segments found. Creating new log.
I20260811 12:37:14.279561 31977 log.cc:824] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:14.280612 31978 log.cc:824] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:14.282157 31974 tablet_bootstrap.cc:492] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: No bootstrap required, opened a new log
I20260811 12:37:14.282642 31974 ts_tablet_manager.cc:1397] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260811 12:37:14.282792 31977 tablet_bootstrap.cc:492] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: No bootstrap required, opened a new log
I20260811 12:37:14.283067 31977 ts_tablet_manager.cc:1397] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260811 12:37:14.283620 31978 tablet_bootstrap.cc:492] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: No bootstrap required, opened a new log
I20260811 12:37:14.283876 31978 ts_tablet_manager.cc:1397] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260811 12:37:14.291055 31978 raft_consensus.cc:348] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.291442 31978 raft_consensus.cc:374] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:37:14.291534 31978 raft_consensus.cc:729] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c04c7984444e42a4db7a19ab042987, State: Initialized, Role: FOLLOWER
I20260811 12:37:14.292207 31978 consensus_queue.cc:260] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.293362 31943 heartbeater.cc:503] Master 127.3.204.126:33833 was elected leader, sending a full tablet report...
I20260811 12:37:14.293303 31974 raft_consensus.cc:348] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.293375 31977 raft_consensus.cc:348] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.293804 31977 raft_consensus.cc:374] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:37:14.293809 31974 raft_consensus.cc:374] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:37:14.293895 31977 raft_consensus.cc:729] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b8bb3c552f4efa82899e5ef437a2b0, State: Initialized, Role: FOLLOWER
I20260811 12:37:14.293906 31974 raft_consensus.cc:729] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5fe9d1f56244f990d8a5e608b13fdf, State: Initialized, Role: FOLLOWER
I20260811 12:37:14.294097 31978 ts_tablet_manager.cc:1428] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260811 12:37:14.294621 31977 consensus_queue.cc:260] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.294723 31974 consensus_queue.cc:260] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.296470 31719 heartbeater.cc:503] Master 127.3.204.126:33833 was elected leader, sending a full tablet report...
I20260811 12:37:14.297508 31974 ts_tablet_manager.cc:1428] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260811 12:37:14.298429 31977 ts_tablet_manager.cc:1428] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260811 12:37:14.334347 31469 tablet.cc:2369] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:37:14.377141 31720 tablet.cc:2369] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:37:14.407780 31944 tablet.cc:2369] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:37:14.490409 31984 raft_consensus.cc:482] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:37:14.490751 31984 raft_consensus.cc:504] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.492578 31984 leader_election.cc:288] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), af5fe9d1f56244f990d8a5e608b13fdf (127.3.204.65:46501)
I20260811 12:37:14.502214 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.493021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:50305 (local address 127.3.204.67:43673)
0811 12:37:14.493238 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.493244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.494170 (+ 926us) server_negotiation.cc:407] Connection header received
0811 12:37:14.494276 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.494282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.494365 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.494470 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:14.495338 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.496522 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.499748 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.501055 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.501230 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.501314 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.501817 (+ 503us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.501958 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 12:37:14.502218 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.493218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:37407 (local address 127.3.204.65:46501)
0811 12:37:14.493428 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.493434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.493983 (+ 549us) server_negotiation.cc:407] Connection header received
0811 12:37:14.494286 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.494292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.494365 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.494470 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:14.495337 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.496519 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.499748 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.501053 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.501232 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.501314 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.501817 (+ 503us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.501973 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:37:14.502693 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.492925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.3.204.66:37407)
0811 12:37:14.493837 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:14.493861 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:14.494052 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:14.494514 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:14.494520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:14.494559 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:14.495169 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.495178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.496673 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.496679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.499522 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.499535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.501352 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.501360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.501596 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.501660 (+ 64us) client_negotiation.cc:769] Sending connection context
0811 12:37:14.501761 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 12:37:14.501926 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":377,"spinlock_wait_cycles":19712,"thread_start_us":493,"threads_started":1}
I20260811 12:37:14.502719 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.493159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43673 (local address 127.3.204.66:50305)
0811 12:37:14.494033 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:14.494048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:14.494150 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:14.494514 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:14.494520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:14.494700 (+ 180us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:14.495158 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.495173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.496675 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.496679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.499514 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:14.499535 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.501360 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.501367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:14.501595 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.501659 (+ 64us) client_negotiation.cc:769] Sending connection context
0811 12:37:14.501761 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 12:37:14.501925 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":134,"spinlock_wait_cycles":43776,"thread_start_us":502,"threads_started":1}
I20260811 12:37:14.504050 31866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076bbd00fc2a42e3a5b9824c91d05620" candidate_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c04c7984444e42a4db7a19ab042987" is_pre_election: true
I20260811 12:37:14.504050 31378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076bbd00fc2a42e3a5b9824c91d05620" candidate_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" is_pre_election: true
I20260811 12:37:14.504719 31378 raft_consensus.cc:2454] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b8bb3c552f4efa82899e5ef437a2b0 in term 0.
I20260811 12:37:14.504719 31866 raft_consensus.cc:2454] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b8bb3c552f4efa82899e5ef437a2b0 in term 0.
I20260811 12:37:14.505488 31513 leader_election.cc:302] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0; no voters:
I20260811 12:37:14.505878 31984 raft_consensus.cc:2781] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:37:14.506039 31984 raft_consensus.cc:482] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:37:14.506145 31984 raft_consensus.cc:3037] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:37:14.508378 31984 raft_consensus.cc:504] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.509027 31984 leader_election.cc:288] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [CANDIDATE]: Term 1 election: Requested vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), af5fe9d1f56244f990d8a5e608b13fdf (127.3.204.65:46501)
I20260811 12:37:14.509527 31866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076bbd00fc2a42e3a5b9824c91d05620" candidate_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c04c7984444e42a4db7a19ab042987"
I20260811 12:37:14.509577 31378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076bbd00fc2a42e3a5b9824c91d05620" candidate_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5fe9d1f56244f990d8a5e608b13fdf"
I20260811 12:37:14.509712 31866 raft_consensus.cc:3037] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:37:14.509791 31378 raft_consensus.cc:3037] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:37:14.511940 31866 raft_consensus.cc:2454] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b8bb3c552f4efa82899e5ef437a2b0 in term 1.
I20260811 12:37:14.512012 31378 raft_consensus.cc:2454] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b8bb3c552f4efa82899e5ef437a2b0 in term 1.
I20260811 12:37:14.512365 31513 leader_election.cc:302] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0; no voters:
I20260811 12:37:14.512691 31984 raft_consensus.cc:2781] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:37:14.513129 31984 raft_consensus.cc:686] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 LEADER]: Becoming Leader. State: Replica: d6b8bb3c552f4efa82899e5ef437a2b0, State: Running, Role: LEADER
I20260811 12:37:14.513635 31984 consensus_queue.cc:237] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:14.518539 31121 catalog_manager.cc:5495] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66). New cstate: current_term: 1 leader_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } health_report { overall_health: UNKNOWN } } }
I20260811 12:37:14.540488 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.534306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.3.204.66:42875)
0811 12:37:14.534509 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.534514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.534535 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:37:14.534625 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.534628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.534703 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.534829 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:37:14.535221 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.536132 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.538053 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.538659 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.538788 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.539343 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:37:14.539867 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:37:14.539986 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.540044 (+ 58us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.540152 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:37:14.548955 31378 raft_consensus.cc:1261] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Refusing update from remote peer d6b8bb3c552f4efa82899e5ef437a2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:37:14.548956 31866 raft_consensus.cc:1261] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 1 FOLLOWER]: Refusing update from remote peer d6b8bb3c552f4efa82899e5ef437a2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:37:14.550549 31991 consensus_queue.cc:1035] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:37:14.550959 31984 consensus_queue.cc:1035] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:37:14.563061 31996 mvcc.cc:205] Tried to move back new op lower bound from 7317306714301296640 to 7317306714171576320. Current Snapshot: MvccSnapshot[applied={T|T < 7317306714301296640}]
I20260811 12:37:14.580580 31124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:37:14.580860 31124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:37:14.584474 31124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Sending DeleteTablet for 3 replicas of tablet 076bbd00fc2a42e3a5b9824c91d05620
I20260811 12:37:14.585727 31605 tablet_service.cc:1514] Processing DeleteTablet for tablet 076bbd00fc2a42e3a5b9824c91d05620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:37:14 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:43362
I20260811 12:37:14.586151 31835 tablet_service.cc:1514] Processing DeleteTablet for tablet 076bbd00fc2a42e3a5b9824c91d05620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:37:14 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:34022
I20260811 12:37:14.586534 31344 tablet_service.cc:1514] Processing DeleteTablet for tablet 076bbd00fc2a42e3a5b9824c91d05620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:37:14 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:49472
I20260811 12:37:14.587003 32007 tablet_replica.cc:330] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: stopping tablet replica
I20260811 12:37:14.587527 32007 raft_consensus.cc:2229] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:37:14.588320 32007 raft_consensus.cc:2258] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:37:14.590723 32007 ts_tablet_manager.cc:1905] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:37:14.591365 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 31226
I20260811 12:37:14.598980 32007 ts_tablet_manager.cc:1918] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:37:14.599320 32007 log.cc:1196] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/076bbd00fc2a42e3a5b9824c91d05620
I20260811 12:37:14.600003 32007 ts_tablet_manager.cc:1939] T 076bbd00fc2a42e3a5b9824c91d05620 P d6b8bb3c552f4efa82899e5ef437a2b0: Deleting consensus metadata
I20260811 12:37:14.602273 31104 catalog_manager.cc:4841] TS d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): tablet 076bbd00fc2a42e3a5b9824c91d05620 (table test-table [id=3139c55cb30f43368a2728819031f4b0]) successfully deleted
I20260811 12:37:14.608721 32011 tablet_replica.cc:330] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: stopping tablet replica
I20260811 12:37:14.610275 32011 raft_consensus.cc:2229] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:37:14.610972 32011 raft_consensus.cc:2258] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:37:14.613972 32011 ts_tablet_manager.cc:1905] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:37:14.621750 32011 ts_tablet_manager.cc:1918] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 11 12:37:14 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:37:14.622505 32011 log.cc:1196] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/076bbd00fc2a42e3a5b9824c91d05620
I20260811 12:37:14.622534 32009 tablet_replica.cc:330] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: stopping tablet replica
I20260811 12:37:14.623124 32011 ts_tablet_manager.cc:1939] T 076bbd00fc2a42e3a5b9824c91d05620 P af5fe9d1f56244f990d8a5e608b13fdf: Deleting consensus metadata
I20260811 12:37:14.623466 32009 raft_consensus.cc:2229] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:37:14.624099 32009 raft_consensus.cc:2258] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:37:14.625478 31098 catalog_manager.cc:4841] TS af5fe9d1f56244f990d8a5e608b13fdf (127.3.204.65:46501): tablet 076bbd00fc2a42e3a5b9824c91d05620 (table test-table [id=3139c55cb30f43368a2728819031f4b0]) successfully deleted
I20260811 12:37:14.626502 32009 ts_tablet_manager.cc:1905] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:37:14.634063 32009 ts_tablet_manager.cc:1918] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:37:14.634330 32009 log.cc:1196] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/076bbd00fc2a42e3a5b9824c91d05620
I20260811 12:37:14.634593 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.592300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.3.204.65:46501)
0811 12:37:14.592516 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.592529 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.609615 (+ 17086us) server_negotiation.cc:407] Connection header received
0811 12:37:14.609858 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.609864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.609989 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.610100 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:14.612645 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.613991 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.616426 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.617179 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.617289 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.626627 (+ 9338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:14.626641 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:14.626657 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:14.626708 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:14.630528 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.631612 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.631622 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.631626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.631868 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.632369 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.632375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.632378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.632973 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:14.633265 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.633960 (+ 695us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.634202 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:37:14.635190 32009 ts_tablet_manager.cc:1939] T 076bbd00fc2a42e3a5b9824c91d05620 P 77c04c7984444e42a4db7a19ab042987: Deleting consensus metadata
I20260811 12:37:14.636106 31459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:14.637411 31100 catalog_manager.cc:4841] TS 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): tablet 076bbd00fc2a42e3a5b9824c91d05620 (table test-table [id=3139c55cb30f43368a2728819031f4b0]) successfully deleted
I20260811 12:37:14.729905 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 31226
I20260811 12:37:14.757416 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 31480
Aug 11 12:37:14 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:37:14.774678 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.758293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.3.204.66:42875)
0811 12:37:14.758484 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.758490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.758512 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:37:14.758633 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.758638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.758719 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.758842 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:14.760180 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.761407 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.762699 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.763380 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.763510 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.768185 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:14.768201 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:14.768218 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:14.768274 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:14.771484 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.772475 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.772484 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.772486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.772638 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.773066 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.773070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.773072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.773488 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:14.773740 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.774167 (+ 427us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.774343 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:37:14.775431 31711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:14.891942 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 31480
I20260811 12:37:14.915978 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 31726
Aug 11 12:37:14 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:37:14.932024 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:14.916806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.3.204.67:43673)
0811 12:37:14.917001 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:14.917007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:14.917065 (+ 58us) server_negotiation.cc:407] Connection header received
0811 12:37:14.917283 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:14.917287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:14.917367 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:14.917455 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:14.918835 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.919559 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.921012 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:14.921405 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:14.921521 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:14.926091 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:14.926108 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:14.926125 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:14.926167 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:14.929063 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.929837 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.929846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.929848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.930024 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:14.930414 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:14.930418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:14.930420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:14.930790 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:14.931090 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:14.931439 (+ 349us) server_negotiation.cc:299] Negotiation successful
0811 12:37:14.931684 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:37:14.932724 31934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:15.021591 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 31726
I20260811 12:37:15.048125 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 31069
I20260811 12:37:15.049378 31176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:15.163077 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 31069
I20260811 12:37:35.190775 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33833
--webserver_interface=127.3.204.126
--webserver_port=33961
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:37:35.545912 32021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:35.546211 32021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:35.546324 32021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:35.553723 32021 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 12:37:35.553829 32021 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 12:37:35.553870 32021 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 12:37:35.553951 32021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:37:35.554026 32021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:35.554121 32021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:35.554179 32021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:37:35.554234 32021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:37:35.564559 32021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:33833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=33961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:35.565936 32021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:35.567355 32021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:35.591256 32031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:35.591243 32030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:35.591677 32038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:35.592047 32021 server_base.cc:1034] running on GCE node
I20260811 12:37:35.593029 32021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:35.594632 32021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:35.595872 32021 hybrid_clock.cc:648] HybridClock initialized: now 1786451855595795 us; error 51 us; skew 500 ppm
I20260811 12:37:35.596360 32021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:35 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:35.600021 32021 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:37:35.601691 32021 webserver.cc:460] Webserver started at http://127.3.204.126:33961/ using document root <none> and password file <none>
I20260811 12:37:35.602478 32021 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:35.602586 32021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:35.609625 32021 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 12:37:35.612737 32053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:35.614300 32021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:37:35.614471 32021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "0a8b307ea77640da90db2b1095ee6275"
format_stamp: "Formatted at 2026-08-11 12:37:12 on dist-test-slave-lgr5"
server_key: "e9400dcc57d2a2e7d773ee42306086f2"
server_key_iv: "704d6f401dc82fa995d5d18f4bf3b8f8"
server_key_version: "encryptionkey@0"
I20260811 12:37:35.615588 32021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:35.638671 32021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:35.643357 32021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:35.643613 32021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:35.662158 32021 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:33833
I20260811 12:37:35.662189 32135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:33833 every 8 connection(s)
I20260811 12:37:35.663790 32021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:37:35.671625 32137 sys_catalog.cc:263] Verifying existing consensus state
I20260811 12:37:35.673489 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 32021
I20260811 12:37:35.674686 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:46501
--local_ip_for_outbound_sockets=127.3.204.65
--tserver_master_addrs=127.3.204.126:33833
--webserver_port=46639
--webserver_interface=127.3.204.65
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:37:35.676463 32137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Bootstrap starting.
I20260811 12:37:35.700865 32137 log.cc:824] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:35.714751 32137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:37:35.715402 32137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Bootstrap complete.
I20260811 12:37:35.725419 32137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:35.726346 32137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8b307ea77640da90db2b1095ee6275, State: Initialized, Role: FOLLOWER
I20260811 12:37:35.727020 32137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:35.727273 32137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:37:35.727427 32137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:37:35.727603 32137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 1 FOLLOWER]: Advancing to term 2
I20260811 12:37:35.731879 32137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:35.732515 32137 leader_election.cc:302] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a8b307ea77640da90db2b1095ee6275; no voters:
I20260811 12:37:35.733209 32137 leader_election.cc:288] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 12:37:35.733376 32141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 12:37:35.734659 32141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 2 LEADER]: Becoming Leader. State: Replica: 0a8b307ea77640da90db2b1095ee6275, State: Running, Role: LEADER
I20260811 12:37:35.735421 32141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:35.735910 32137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:37:35.738337 32144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8b307ea77640da90db2b1095ee6275. Latest consensus state: current_term: 2 leader_uuid: "0a8b307ea77640da90db2b1095ee6275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } } }
I20260811 12:37:35.738727 32144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: This master's current role is: LEADER
I20260811 12:37:35.738466 32142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a8b307ea77640da90db2b1095ee6275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } } }
I20260811 12:37:35.738914 32142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: This master's current role is: LEADER
I20260811 12:37:35.748942 32151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:37:35.758668 32151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:37:35.760571 32151 catalog_manager.cc:1259] Loaded cluster ID: 988595181862414597631268084daf5f
I20260811 12:37:35.760761 32151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:37:35.765082 32151 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:37:35.766884 32151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Loaded TSK: 0
I20260811 12:37:35.802395 32151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Generated new TSK 1
I20260811 12:37:35.802848 32151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 12:37:36.042759 32139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:36.043087 32139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:36.043224 32139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:36.050910 32139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:36.051051 32139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:36.051208 32139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:37:36.061553 32139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:46501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=46639
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:36.062978 32139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:36.064479 32139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:36.079895 32179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:36.080446 32182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:36.080480 32178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:36.080919 32139 server_base.cc:1034] running on GCE node
I20260811 12:37:36.081596 32139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:36.082630 32139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:36.083860 32139 hybrid_clock.cc:648] HybridClock initialized: now 1786451856083773 us; error 64 us; skew 500 ppm
I20260811 12:37:36.084271 32139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:36 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:36.087772 32139 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:37:36.089488 32139 webserver.cc:460] Webserver started at http://127.3.204.65:46639/ using document root <none> and password file <none>
I20260811 12:37:36.090240 32139 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:36.090389 32139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:36.096549 32139 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 12:37:36.099586 32190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:36.101275 32139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:37:36.101475 32139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "af5fe9d1f56244f990d8a5e608b13fdf"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "be2d5fa4ccdc6d947021af99aabea016"
server_key_iv: "311f9914176b63f025eabb9a7e4ddb99"
server_key_version: "encryptionkey@0"
I20260811 12:37:36.102589 32139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:36.122788 32139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:36.127539 32139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:36.127811 32139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:36.129038 32139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:37:36.130666 32139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:37:36.130774 32139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:36.130901 32139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:37:36.130975 32139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:36.164222 32139 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:46501
I20260811 12:37:36.164296 32379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:46501 every 8 connection(s)
I20260811 12:37:36.165782 32139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:37:36.168740 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 32139
I20260811 12:37:36.169576 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:42875
--local_ip_for_outbound_sockets=127.3.204.66
--tserver_master_addrs=127.3.204.126:33833
--webserver_port=44809
--webserver_interface=127.3.204.66
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:37:36 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:37:36.192288 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:36.168952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:59343 (local address 127.3.204.126:33833)
0811 12:37:36.169657 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:36.169974 (+ 317us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:36.170242 (+ 268us) server_negotiation.cc:407] Connection header received
0811 12:37:36.172978 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:36.172991 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:36.173331 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:36.173635 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:36.175649 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:36.177004 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:36.179177 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:36.179893 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:36.180036 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:36.184225 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:36.184235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:36.184251 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:36.184301 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:36.187508 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:36.188536 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:36.188544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:36.188551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:36.188744 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:36.189248 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:36.189253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:36.189255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:36.189775 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:36.189971 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:36.190899 (+ 928us) server_negotiation.cc:299] Negotiation successful
0811 12:37:36.191250 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":274,"threads_started":1}
I20260811 12:37:36.192538 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:36.169067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:59343)
0811 12:37:36.170048 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:36.170105 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:36.172377 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:36.173874 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:36.173886 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:36.174621 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:37:36.175427 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:36.175455 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:36.177188 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:36.177193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:36.179005 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:36.179022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:36.180187 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:36.180196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:36.180465 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:36.181184 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:37:36.181218 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:37:36.183891 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:37:36.187674 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:36.187684 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:36.187701 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:36.188394 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:36.188875 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:36.188881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:36.188885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:36.189130 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:36.189941 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:37:36.189950 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:37:36.190378 (+ 428us) client_negotiation.cc:769] Sending connection context
0811 12:37:36.190680 (+ 302us) client_negotiation.cc:240] Negotiation successful
0811 12:37:36.191276 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":453,"threads_started":1}
I20260811 12:37:36.195838 32380 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33833
I20260811 12:37:36.196246 32380 heartbeater.cc:464] Registering TS with master...
I20260811 12:37:36.197253 32380 heartbeater.cc:511] Master 127.3.204.126:33833 requested a full tablet report, sending...
I20260811 12:37:36.199936 32076 ts_manager.cc:194] Registered new tserver with Master: af5fe9d1f56244f990d8a5e608b13fdf (127.3.204.65:46501)
I20260811 12:37:36.201563 32076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:59343
W20260811 12:37:36.525276 32393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:36.525592 32393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:36.525718 32393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:36.533147 32393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:36.533280 32393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:36.533425 32393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:37:36.543682 32393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:42875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=44809
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:36.545087 32393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:36.546366 32393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:36.559597 32405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:36.559620 32409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:36.559674 32414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:36.560222 32393 server_base.cc:1034] running on GCE node
I20260811 12:37:36.560920 32393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:36.561861 32393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:36.563061 32393 hybrid_clock.cc:648] HybridClock initialized: now 1786451856563007 us; error 46 us; skew 500 ppm
I20260811 12:37:36.563382 32393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:36 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:36.570397 32393 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:37:36.572103 32393 webserver.cc:460] Webserver started at http://127.3.204.66:44809/ using document root <none> and password file <none>
I20260811 12:37:36.572837 32393 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:36.572962 32393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:36.578951 32393 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 12:37:36.582283 32425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:36.583777 32393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:37:36.583963 32393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "d6b8bb3c552f4efa82899e5ef437a2b0"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "dd7960f815a9b1786c4d11fe20c84bb4"
server_key_iv: "09b5c73d4fb89773416b5e92b1594f95"
server_key_version: "encryptionkey@0"
I20260811 12:37:36.585116 32393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:36.604673 32393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:36.609722 32393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:36.609997 32393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:36.611022 32393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:37:36.612461 32393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:37:36.612563 32393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:36.612675 32393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:37:36.612747 32393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:36.645248 32393 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:42875
I20260811 12:37:36.645318 32640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:42875 every 8 connection(s)
I20260811 12:37:36.646740 32393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:37:36.656452 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 32393
I20260811 12:37:36.657143 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:43673
--local_ip_for_outbound_sockets=127.3.204.67
--tserver_master_addrs=127.3.204.126:33833
--webserver_port=41053
--webserver_interface=127.3.204.67
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 12:37:36 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:37:36.668903 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:36.649710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:57043 (local address 127.3.204.126:33833)
0811 12:37:36.649920 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:36.649927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:36.651089 (+ 1162us) server_negotiation.cc:407] Connection header received
0811 12:37:36.652914 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:36.652918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:36.652987 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:36.653065 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:36.654621 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:36.655347 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:36.657029 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:36.657654 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:36.657832 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:36.661596 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:36.661604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:36.661609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:36.661656 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:36.665051 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:36.666088 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:36.666094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:36.666098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:36.666254 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:36.666774 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:36.666779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:36.666783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:36.667168 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:36.667329 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:36.668253 (+ 924us) server_negotiation.cc:299] Negotiation successful
0811 12:37:36.668563 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:37:36.670059 32650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:36.650190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:57043)
0811 12:37:36.650924 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:36.650963 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:36.652653 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:36.653327 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:36.653336 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:36.653881 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:37:36.654433 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:36.654461 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:36.655508 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:36.655511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:36.656876 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:36.656889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:36.657957 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:36.657963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:36.658169 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:36.658869 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:37:36.658897 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:37:36.661305 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:37:36.665234 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:36.665245 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:36.665261 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:36.665902 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:36.666374 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:36.666379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:36.666382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:36.666638 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:36.667315 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:37:36.667323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:37:36.667772 (+ 449us) client_negotiation.cc:769] Sending connection context
0811 12:37:36.668119 (+ 347us) client_negotiation.cc:240] Negotiation successful
0811 12:37:36.668664 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":290,"threads_started":1}
I20260811 12:37:36.671780 32641 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33833
I20260811 12:37:36.672187 32641 heartbeater.cc:464] Registering TS with master...
I20260811 12:37:36.673121 32641 heartbeater.cc:511] Master 127.3.204.126:33833 requested a full tablet report, sending...
I20260811 12:37:36.674958 32076 ts_manager.cc:194] Registered new tserver with Master: d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
I20260811 12:37:36.676074 32076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:57043
W20260811 12:37:37.008169 32651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:37.008503 32651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:37.008659 32651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:37.016306 32651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:37.016443 32651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:37.016611 32651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:37:37.026984 32651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:43673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=41053
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:37.028455 32651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:37.029940 32651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:37.044137 32667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:37.044137 32661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:37.044030 32660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:37.045358 32651 server_base.cc:1034] running on GCE node
I20260811 12:37:37.045974 32651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:37.046945 32651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:37.048211 32651 hybrid_clock.cc:648] HybridClock initialized: now 1786451857048162 us; error 31 us; skew 500 ppm
I20260811 12:37:37.048593 32651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:37 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451857, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:37.051783 32651 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:37:37.053334 32651 webserver.cc:460] Webserver started at http://127.3.204.67:41053/ using document root <none> and password file <none>
I20260811 12:37:37.054032 32651 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:37.054175 32651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:37.059762 32651 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260811 12:37:37.063042 32683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:37.064673 32651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 12:37:37.064865 32651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "77c04c7984444e42a4db7a19ab042987"
format_stamp: "Formatted at 2026-08-11 12:37:14 on dist-test-slave-lgr5"
server_key: "99e24c6be1313120843dce135a5b2699"
server_key_iv: "0135bcdc61fd18efc3be29f06b9a222f"
server_key_version: "encryptionkey@0"
I20260811 12:37:37.065922 32651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:37.082780 32651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:37.087509 32651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:37.087798 32651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:37.088907 32651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:37:37.090422 32651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:37:37.090528 32651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:37.090642 32651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:37:37.090719 32651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:37.125746 32651 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:43673
I20260811 12:37:37.125808 438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:43673 every 8 connection(s)
I20260811 12:37:37.127280 32651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:37:37.132295 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 32651
Aug 11 12:37:37 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451857, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:37:37.152666 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.133281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.3.204.126:33833)
0811 12:37:37.133812 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.133819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.134425 (+ 606us) server_negotiation.cc:407] Connection header received
0811 12:37:37.134631 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.134635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.134696 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.134769 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:37.137229 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.138296 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.140347 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.141140 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.141358 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.144889 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:37.144895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:37.144899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:37.144938 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:37.148658 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.149598 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.149603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.149607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.149809 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.150360 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.150366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.150369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.150747 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:37.150993 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.152074 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.152322 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":384,"spinlock_wait_cycles":3712,"thread_start_us":265,"threads_started":1}
I20260811 12:37:37.153280 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.130017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:34991 (local address 127.3.204.126:33833)
0811 12:37:37.130218 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.130223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.131677 (+ 1454us) server_negotiation.cc:407] Connection header received
0811 12:37:37.134134 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.134140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.134220 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.134311 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:37.136714 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.137891 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.140347 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.141140 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.141378 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.145572 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:37.145579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:37.145584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:37.145624 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:37.149053 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.149983 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.149990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.149993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.150137 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.150775 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.150779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.150781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.151153 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:37.151307 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.152155 (+ 848us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.152498 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":234,"server-negotiator.queue_time_us":72,"spinlock_wait_cycles":11264}
I20260811 12:37:37.154040 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.130496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:34991)
0811 12:37:37.131441 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.131495 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:37.133724 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:37.134650 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:37.134659 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:37.135517 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:37:37.136506 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.136540 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.138090 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.138095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.140157 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.140174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.141395 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.141403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.141722 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.142344 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:37:37.142376 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:37:37.145300 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:37:37.149225 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:37.149235 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:37.149251 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:37.149849 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:37.150300 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:37:37.150305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:37:37.150309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:37:37.150631 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:37:37.151304 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:37:37.151312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:37:37.151729 (+ 417us) client_negotiation.cc:769] Sending connection context
0811 12:37:37.151998 (+ 269us) client_negotiation.cc:240] Negotiation successful
0811 12:37:37.152606 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":367,"threads_started":1}
I20260811 12:37:37.155511 441 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33833
I20260811 12:37:37.155853 441 heartbeater.cc:464] Registering TS with master...
I20260811 12:37:37.156649 441 heartbeater.cc:511] Master 127.3.204.126:33833 requested a full tablet report, sending...
I20260811 12:37:37.158149 32076 ts_manager.cc:194] Registered new tserver with Master: 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673)
I20260811 12:37:37.159025 32076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:34991
I20260811 12:37:37.168864 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:37:37.175917 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.170080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.3.204.126:33833)
0811 12:37:37.170256 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.170261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.170656 (+ 395us) server_negotiation.cc:407] Connection header received
0811 12:37:37.170847 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.170851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.171015 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.171149 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:37:37.171574 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.172462 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.174012 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.174484 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.174586 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.175093 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:37:37.175378 (+ 285us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0811 12:37:37.175718 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49100: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260811 12:37:37.176031 32389 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49100: token expired
W20260811 12:37:37.176666 31954 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.204.126:33833)
I20260811 12:37:37.190207 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.177301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.3.204.126:33833)
0811 12:37:37.177459 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.177463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.177477 (+ 14us) server_negotiation.cc:407] Connection header received
0811 12:37:37.177517 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.177520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.177577 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.177648 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:37.179085 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.179896 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.181436 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.181850 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.181946 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.184802 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:37.184809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:37.184812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:37.184851 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:37.187384 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.188046 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.188050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.188052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.188212 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.188598 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.188603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.188606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.189029 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:37.189262 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.189635 (+ 373us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.189873 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 12:37:37.193943 32076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:37:37.201757 32076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:37:37.205667 32380 heartbeater.cc:503] Master 127.3.204.126:33833 was elected leader, sending a full tablet report...
I20260811 12:37:37.238050 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.214714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.3.204.66:42875)
0811 12:37:37.227168 (+ 12454us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.227181 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.228947 (+ 1766us) server_negotiation.cc:407] Connection header received
0811 12:37:37.229489 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.229498 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.229771 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.230014 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:37.231206 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.232338 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.234889 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.236170 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.236307 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.236518 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.237647 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.237790 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12184,"thread_start_us":12481,"threads_started":1}
I20260811 12:37:37.238657 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.214254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:42875 (local address 127.0.0.1:44100)
0811 12:37:37.228656 (+ 14402us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.228758 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:37.229156 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:37.230316 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:37.230320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:37.230382 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:37.230996 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.231012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.232527 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.232532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.234721 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.234734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.236756 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.236772 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.236996 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.237051 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 12:37:37.237193 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 12:37:37.237404 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14180,"mutex_wait_us":1,"spinlock_wait_cycles":136960,"thread_start_us":14686,"threads_started":1}
I20260811 12:37:37.239197 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.216398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:37954)
0811 12:37:37.229082 (+ 12684us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.229104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:37.229209 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:37.230298 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:37.230306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:37.230357 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:37.230851 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.230864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.232904 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.232909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.235512 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.235527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.238396 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.238403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.238539 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.238564 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 12:37:37.238664 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 12:37:37.238812 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12560,"mutex_wait_us":26,"spinlock_wait_cycles":1792,"thread_start_us":12076,"threads_started":1}
I20260811 12:37:37.239391 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.214421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.3.204.67:43673)
0811 12:37:37.227998 (+ 13577us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.228008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.229523 (+ 1515us) server_negotiation.cc:407] Connection header received
0811 12:37:37.229830 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.229838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.230052 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.230205 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:37.231312 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.232918 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.235704 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.237045 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.237177 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.237411 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.238870 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.239068 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13404,"thread_start_us":13241,"threads_started":1}
I20260811 12:37:37.240037 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.214157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43673 (local address 127.0.0.1:48066)
0811 12:37:37.228656 (+ 14499us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.229433 (+ 777us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:37.229630 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:37.230535 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:37.230539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:37.230585 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:37.231078 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.231090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.233067 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.233072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.235512 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.235527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.238437 (+ 2910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.238452 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.238652 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.238684 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 12:37:37.239538 (+ 854us) client_negotiation.cc:240] Negotiation successful
0811 12:37:37.239674 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14155,"mutex_wait_us":89,"thread_start_us":14500,"threads_started":1}
I20260811 12:37:37.243144 32522 tablet_service.cc:1467] Processing CreateTablet for tablet 818ff82eeff74fd7898eea6bd2af6573 (DEFAULT_TABLE table=test-table [id=df6d1c7cc80c437488389dc3b7929f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:37:37.239876 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.217073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.3.204.65:46501)
0811 12:37:37.228593 (+ 11520us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.228601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.229398 (+ 797us) server_negotiation.cc:407] Connection header received
0811 12:37:37.229514 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.229522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.229783 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.229975 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:37.231091 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.232737 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.235690 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.237129 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.237250 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.237477 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.238945 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.239140 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11341,"thread_start_us":11170,"threads_started":1}
I20260811 12:37:37.243754 317 tablet_service.cc:1467] Processing CreateTablet for tablet 818ff82eeff74fd7898eea6bd2af6573 (DEFAULT_TABLE table=test-table [id=df6d1c7cc80c437488389dc3b7929f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:37:37.244856 32522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818ff82eeff74fd7898eea6bd2af6573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:37:37.245213 317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818ff82eeff74fd7898eea6bd2af6573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:37:37.247531 32272 tablet_service.cc:1467] Processing CreateTablet for tablet 818ff82eeff74fd7898eea6bd2af6573 (DEFAULT_TABLE table=test-table [id=df6d1c7cc80c437488389dc3b7929f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:37:37.248747 32272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818ff82eeff74fd7898eea6bd2af6573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:37:37.261438 465 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Bootstrap starting.
I20260811 12:37:37.264922 465 tablet_bootstrap.cc:654] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Neither blocks nor log segments found. Creating new log.
I20260811 12:37:37.266053 465 log.cc:824] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:37.266971 466 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Bootstrap starting.
I20260811 12:37:37.267222 468 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Bootstrap starting.
I20260811 12:37:37.270889 468 tablet_bootstrap.cc:654] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Neither blocks nor log segments found. Creating new log.
I20260811 12:37:37.270985 466 tablet_bootstrap.cc:654] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Neither blocks nor log segments found. Creating new log.
I20260811 12:37:37.272141 468 log.cc:824] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:37.272272 465 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: No bootstrap required, opened a new log
I20260811 12:37:37.272342 466 log.cc:824] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:37.272574 465 ts_tablet_manager.cc:1397] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260811 12:37:37.275440 468 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: No bootstrap required, opened a new log
I20260811 12:37:37.275449 466 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: No bootstrap required, opened a new log
I20260811 12:37:37.275724 466 ts_tablet_manager.cc:1397] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260811 12:37:37.275718 468 ts_tablet_manager.cc:1397] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 12:37:37.283385 465 raft_consensus.cc:348] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.283924 465 raft_consensus.cc:374] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:37:37.283689 466 raft_consensus.cc:348] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.284031 465 raft_consensus.cc:729] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5fe9d1f56244f990d8a5e608b13fdf, State: Initialized, Role: FOLLOWER
I20260811 12:37:37.284106 466 raft_consensus.cc:374] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:37:37.284211 466 raft_consensus.cc:729] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b8bb3c552f4efa82899e5ef437a2b0, State: Initialized, Role: FOLLOWER
I20260811 12:37:37.284818 466 consensus_queue.cc:260] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.284804 465 consensus_queue.cc:260] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.286052 32641 heartbeater.cc:503] Master 127.3.204.126:33833 was elected leader, sending a full tablet report...
I20260811 12:37:37.286273 468 raft_consensus.cc:348] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.286722 466 ts_tablet_manager.cc:1428] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260811 12:37:37.286731 468 raft_consensus.cc:374] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:37:37.286825 468 raft_consensus.cc:729] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c04c7984444e42a4db7a19ab042987, State: Initialized, Role: FOLLOWER
I20260811 12:37:37.287456 468 consensus_queue.cc:260] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.288945 465 ts_tablet_manager.cc:1428] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260811 12:37:37.289904 441 heartbeater.cc:503] Master 127.3.204.126:33833 was elected leader, sending a full tablet report...
I20260811 12:37:37.291741 468 ts_tablet_manager.cc:1428] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260811 12:37:37.341019 476 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:37:37.341380 476 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.343158 476 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875), af5fe9d1f56244f990d8a5e608b13fdf (127.3.204.65:46501)
I20260811 12:37:37.349895 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.343698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:48667 (local address 127.3.204.66:42875)
0811 12:37:37.343960 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.343967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.344064 (+ 97us) server_negotiation.cc:407] Connection header received
0811 12:37:37.344323 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.344327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.344391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.344468 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:37.344975 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.345985 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.347935 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.348986 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.349117 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.349174 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.349530 (+ 356us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.349632 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 12:37:37.349959 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.343707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:42875 (local address 127.3.204.67:48667)
0811 12:37:37.343951 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.343973 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:37.344143 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:37.344533 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:37.344537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:37.344574 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:37.344787 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.344806 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.346145 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.346151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.347756 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.347769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.349146 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.349152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.349346 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.349396 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 12:37:37.349498 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 12:37:37.349636 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260811 12:37:37.350636 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.343685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:35581 (local address 127.3.204.65:46501)
0811 12:37:37.344150 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.344157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.344725 (+ 568us) server_negotiation.cc:407] Connection header received
0811 12:37:37.344855 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.344860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.344942 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.345014 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:37:37.345686 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.346660 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.348651 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.349689 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.349840 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.349921 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.350255 (+ 334us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.350386 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260811 12:37:37.350680 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.343739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.3.204.67:35581)
0811 12:37:37.344613 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.344630 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:37:37.344741 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:37:37.345087 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:37:37.345092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:37:37.345119 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:37:37.345487 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.345501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.346818 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.346822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.348452 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:37:37.348468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.349894 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.349900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:37:37.350095 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.350128 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 12:37:37.350219 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 12:37:37.350344 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":528,"threads_started":1}
I20260811 12:37:37.351146 32562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818ff82eeff74fd7898eea6bd2af6573" candidate_uuid: "77c04c7984444e42a4db7a19ab042987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" is_pre_election: true
I20260811 12:37:37.351755 32562 raft_consensus.cc:2454] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c04c7984444e42a4db7a19ab042987 in term 0.
I20260811 12:37:37.351717 32306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818ff82eeff74fd7898eea6bd2af6573" candidate_uuid: "77c04c7984444e42a4db7a19ab042987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" is_pre_election: true
I20260811 12:37:37.352252 32306 raft_consensus.cc:2454] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c04c7984444e42a4db7a19ab042987 in term 0.
I20260811 12:37:37.352617 32696 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0; no voters:
I20260811 12:37:37.353040 476 raft_consensus.cc:2781] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:37:37.353216 476 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:37:37.353325 476 raft_consensus.cc:3037] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:37:37.355984 476 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.356698 476 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [CANDIDATE]: Term 1 election: Requested vote from peers d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875), af5fe9d1f56244f990d8a5e608b13fdf (127.3.204.65:46501)
I20260811 12:37:37.357163 32562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818ff82eeff74fd7898eea6bd2af6573" candidate_uuid: "77c04c7984444e42a4db7a19ab042987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0"
I20260811 12:37:37.357275 32306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818ff82eeff74fd7898eea6bd2af6573" candidate_uuid: "77c04c7984444e42a4db7a19ab042987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5fe9d1f56244f990d8a5e608b13fdf"
I20260811 12:37:37.357362 32562 raft_consensus.cc:3037] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:37:37.357518 32306 raft_consensus.cc:3037] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:37:37.359712 32306 raft_consensus.cc:2454] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c04c7984444e42a4db7a19ab042987 in term 1.
I20260811 12:37:37.359716 32562 raft_consensus.cc:2454] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c04c7984444e42a4db7a19ab042987 in term 1.
I20260811 12:37:37.360162 32688 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c04c7984444e42a4db7a19ab042987, af5fe9d1f56244f990d8a5e608b13fdf; no voters:
I20260811 12:37:37.360461 476 raft_consensus.cc:2781] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:37:37.360924 476 raft_consensus.cc:686] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 1 LEADER]: Becoming Leader. State: Replica: 77c04c7984444e42a4db7a19ab042987, State: Running, Role: LEADER
I20260811 12:37:37.361398 476 consensus_queue.cc:237] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:37.366219 32076 catalog_manager.cc:5495] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77c04c7984444e42a4db7a19ab042987 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "77c04c7984444e42a4db7a19ab042987" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } health_report { overall_health: UNKNOWN } } }
W20260811 12:37:37.378990 442 tablet.cc:2369] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:37:37.398072 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.392405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.3.204.67:43673)
0811 12:37:37.392619 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.392626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.392645 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:37:37.392884 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.392888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.392980 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.393121 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:37:37.393511 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.394265 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.395698 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.396174 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.396295 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.396886 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:37:37.397433 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:37:37.397595 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.397700 (+ 105us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.397797 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260811 12:37:37.398926 32647 tablet.cc:2369] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:37:37.406567 32306 raft_consensus.cc:1261] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Refusing update from remote peer 77c04c7984444e42a4db7a19ab042987: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:37:37.406766 32562 raft_consensus.cc:1261] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 FOLLOWER]: Refusing update from remote peer 77c04c7984444e42a4db7a19ab042987: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:37:37.407698 479 consensus_queue.cc:1035] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:37:37.408100 476 consensus_queue.cc:1035] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 12:37:37.419368 32387 tablet.cc:2369] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:37:37.421468 484 mvcc.cc:205] Tried to move back new op lower bound from 7317306807927599104 to 7317306807756509184. Current Snapshot: MvccSnapshot[applied={T|T < 7317306807927599104}]
I20260811 12:37:37.434798 32073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:37:37.435104 32073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:37:37.439157 32073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Sending DeleteTablet for 3 replicas of tablet 818ff82eeff74fd7898eea6bd2af6573
I20260811 12:37:37.440529 317 tablet_service.cc:1514] Processing DeleteTablet for tablet 818ff82eeff74fd7898eea6bd2af6573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:37:37 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:48066
I20260811 12:37:37.440624 32522 tablet_service.cc:1514] Processing DeleteTablet for tablet 818ff82eeff74fd7898eea6bd2af6573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:37:37 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:44100
I20260811 12:37:37.440734 32272 tablet_service.cc:1514] Processing DeleteTablet for tablet 818ff82eeff74fd7898eea6bd2af6573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:37:37 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:37954
I20260811 12:37:37.441923 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 32139
I20260811 12:37:37.453088 501 tablet_replica.cc:330] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: stopping tablet replica
I20260811 12:37:37.453797 501 raft_consensus.cc:2229] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:37:37.454754 501 raft_consensus.cc:2258] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:37:37.455713 500 tablet_replica.cc:330] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: stopping tablet replica
I20260811 12:37:37.457904 501 ts_tablet_manager.cc:1905] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:37:37.458498 500 raft_consensus.cc:2229] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:37:37.464658 500 raft_consensus.cc:2258] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:37:37.465075 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.443184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.3.204.65:46501)
0811 12:37:37.443803 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.443810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.443829 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:37:37.443889 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.443895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.443998 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.444108 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:37.446159 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.447496 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.449812 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.450524 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.450645 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.456343 (+ 5698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:37.456362 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:37.456382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:37.456428 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:37.460603 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.461603 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.461612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.461616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.461841 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.462296 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.462301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.462304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.462803 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:37.463107 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.464441 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.464680 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484}
I20260811 12:37:37.465924 32370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:37.467319 500 ts_tablet_manager.cc:1905] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:37:37.469122 501 ts_tablet_manager.cc:1918] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:37:37.469396 501 log.cc:1196] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/818ff82eeff74fd7898eea6bd2af6573
I20260811 12:37:37.470012 501 ts_tablet_manager.cc:1939] T 818ff82eeff74fd7898eea6bd2af6573 P 77c04c7984444e42a4db7a19ab042987: Deleting consensus metadata
I20260811 12:37:37.472147 32056 catalog_manager.cc:4841] TS 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): tablet 818ff82eeff74fd7898eea6bd2af6573 (table test-table [id=df6d1c7cc80c437488389dc3b7929f05]) successfully deleted
I20260811 12:37:37.476014 500 ts_tablet_manager.cc:1918] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:37:37.476213 500 log.cc:1196] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/818ff82eeff74fd7898eea6bd2af6573
I20260811 12:37:37.476750 500 ts_tablet_manager.cc:1939] T 818ff82eeff74fd7898eea6bd2af6573 P d6b8bb3c552f4efa82899e5ef437a2b0: Deleting consensus metadata
I20260811 12:37:37.478817 32061 catalog_manager.cc:4841] TS d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): tablet 818ff82eeff74fd7898eea6bd2af6573 (table test-table [id=df6d1c7cc80c437488389dc3b7929f05]) successfully deleted
I20260811 12:37:37.562645 504 tablet_replica.cc:330] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: stopping tablet replica
I20260811 12:37:37.563097 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 32139
W20260811 12:37:37.595170 32054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.65:46501 (error 108)
I20260811 12:37:37.595825 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 32393
I20260811 12:37:37.597558 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.597050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:37962)
0811 12:37:37.597238 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.597366 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260811 12:37:37.597829 32054 catalog_manager.cc:4584] TS af5fe9d1f56244f990d8a5e608b13fdf (127.3.204.65:46501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 818ff82eeff74fd7898eea6bd2af6573: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
I20260811 12:37:37.612669 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.596783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.3.204.66:42875)
0811 12:37:37.596988 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.596993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.597013 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:37:37.597216 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.597221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.597306 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.597442 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:37.599269 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.600023 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.601921 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.602355 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.602450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.605853 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:37.605867 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:37.605883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:37.605916 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:37.608753 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.609634 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.609644 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.609647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.609811 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.610257 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.610263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.610267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.610867 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:37.611180 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.611819 (+ 639us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.612069 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 12:37:37.613454 32635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:37.651643 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.651075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:37964)
0811 12:37:37.651306 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.651439 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260811 12:37:37.652982 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.652540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:37970)
0811 12:37:37.652709 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.652828 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 12:37:37.713806 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.713203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:37972)
0811 12:37:37.713472 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.713593 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260811 12:37:37.715960 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.714700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:37978)
0811 12:37:37.715691 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.715803 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":869}
I20260811 12:37:37.716493 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 32393
I20260811 12:37:37.748557 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 32651
I20260811 12:37:37.765144 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.749451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.3.204.67:43673)
0811 12:37:37.749656 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 12:37:37.749667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:37:37.749691 (+ 24us) server_negotiation.cc:407] Connection header received
0811 12:37:37.749870 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:37:37.749876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:37:37.749968 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:37:37.750094 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:37:37.751583 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.752854 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.754208 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:37:37.754904 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:37:37.755074 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:37:37.757789 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:37:37.757809 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:37:37.757833 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:37:37.757890 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:37:37.761938 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.762748 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.762760 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.762772 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.762988 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:37:37.763360 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:37:37.763366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:37:37.763369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:37:37.763863 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:37:37.764215 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0811 12:37:37.764480 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 12:37:37.764722 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 12:37:37.765933 433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:37.797313 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.796805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:37986)
0811 12:37:37.797045 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.797146 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260811 12:37:37.798552 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:37.798092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:38000)
0811 12:37:37.798332 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:37.798415 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260811 12:37:37.892359 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 32651
I20260811 12:37:37.922016 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 32021
I20260811 12:37:37.923278 32129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:37:38.040413 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:38.039735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46501 (local address 127.0.0.1:38004)
0811 12:37:38.039976 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:38.040119 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260811 12:37:38.040544 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 32021
I20260811 12:37:58.071722 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33833
--webserver_interface=127.3.204.126
--webserver_port=33961
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:37:58.430943 515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:58.431267 515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:58.431370 515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:58.438879 515 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 12:37:58.438988 515 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 12:37:58.439059 515 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 12:37:58.439131 515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:37:58.439201 515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:58.439299 515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:58.439357 515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:37:58.439436 515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:37:58.449812 515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:33833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=33961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:58.451189 515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:58.452592 515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:58.465148 529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:58.465178 526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:58.465379 525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:58.465914 515 server_base.cc:1034] running on GCE node
I20260811 12:37:58.466892 515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:58.468501 515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:58.469744 515 hybrid_clock.cc:648] HybridClock initialized: now 1786451878469687 us; error 35 us; skew 500 ppm
I20260811 12:37:58.470103 515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:58 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451878, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:58.477463 515 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:37:58.479238 515 webserver.cc:460] Webserver started at http://127.3.204.126:33961/ using document root <none> and password file <none>
I20260811 12:37:58.480042 515 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:58.480214 515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:58.486624 515 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 12:37:58.490330 542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:58.492038 515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:37:58.492309 515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "0a8b307ea77640da90db2b1095ee6275"
format_stamp: "Formatted at 2026-08-11 12:37:12 on dist-test-slave-lgr5"
server_key: "e9400dcc57d2a2e7d773ee42306086f2"
server_key_iv: "704d6f401dc82fa995d5d18f4bf3b8f8"
server_key_version: "encryptionkey@0"
I20260811 12:37:58.493393 515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:58.505765 515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:58.510185 515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:58.510476 515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:58.529350 515 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:33833
I20260811 12:37:58.529372 621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:33833 every 8 connection(s)
I20260811 12:37:58.530939 515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:37:58.535173 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 515
I20260811 12:37:58.536029 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:46501
--local_ip_for_outbound_sockets=127.3.204.65
--tserver_master_addrs=127.3.204.126:33833
--webserver_port=46639
--webserver_interface=127.3.204.65
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:37:58.538467 622 sys_catalog.cc:263] Verifying existing consensus state
I20260811 12:37:58.541870 622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Bootstrap starting.
I20260811 12:37:58.563009 622 log.cc:824] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:58.580065 622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:37:58.580667 622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Bootstrap complete.
I20260811 12:37:58.590644 622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:58.591622 622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8b307ea77640da90db2b1095ee6275, State: Initialized, Role: FOLLOWER
I20260811 12:37:58.592346 622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:58.592594 622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:37:58.592734 622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:37:58.592919 622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 2 FOLLOWER]: Advancing to term 3
I20260811 12:37:58.597159 622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:58.597724 622 leader_election.cc:302] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a8b307ea77640da90db2b1095ee6275; no voters:
I20260811 12:37:58.598431 622 leader_election.cc:288] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 12:37:58.598623 626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 12:37:58.599782 626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [term 3 LEADER]: Becoming Leader. State: Replica: 0a8b307ea77640da90db2b1095ee6275, State: Running, Role: LEADER
I20260811 12:37:58.600443 626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } }
I20260811 12:37:58.600796 622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:37:58.602989 627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0a8b307ea77640da90db2b1095ee6275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } } }
I20260811 12:37:58.603449 627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: This master's current role is: LEADER
I20260811 12:37:58.604202 628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8b307ea77640da90db2b1095ee6275. Latest consensus state: current_term: 3 leader_uuid: "0a8b307ea77640da90db2b1095ee6275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8b307ea77640da90db2b1095ee6275" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33833 } } }
I20260811 12:37:58.604557 628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275 [sys.catalog]: This master's current role is: LEADER
I20260811 12:37:58.613641 634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:37:58.627568 634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:37:58.629643 634 catalog_manager.cc:1259] Loaded cluster ID: 988595181862414597631268084daf5f
I20260811 12:37:58.629906 634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:37:58.634486 634 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:37:58.635944 634 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Loaded TSK: 0
I20260811 12:37:58.636123 634 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a8b307ea77640da90db2b1095ee6275: Loaded TSK: 1
F20260811 12:37:58.637059 634 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f260daac9ad google::LogMessage::Fail() at ??:0
@ 0x7f260daaf88a google::LogMessage::SendToLog() at ??:0
@ 0x7f260daac4de google::LogMessage::Flush() at ??:0
@ 0x7f260daada19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f26118107d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f26117ff73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f26234df1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f26234de783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f262356a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f262356a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f26235b45e9 std::function<>::operator()() at ??:0
@ 0x7f26234e2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f26234e0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2623568610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2623568439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f26106f8345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f26106fb91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f26106fb799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f26106d6eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f26139b06db start_thread at ??:0
@ 0x7f260c94661f clone at ??:0
W20260811 12:37:58.963510 624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:58.963857 624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:58.964017 624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:58.971786 624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:58.971962 624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:58.972155 624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:37:58.982518 624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:46501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=46639
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:58.984194 624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:58.985862 624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:58.999945 696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:58.999878 699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:58.999961 692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:59.000780 624 server_base.cc:1034] running on GCE node
I20260811 12:37:59.001443 624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:59.002457 624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:59.003679 624 hybrid_clock.cc:648] HybridClock initialized: now 1786451879003613 us; error 56 us; skew 500 ppm
I20260811 12:37:59.004024 624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:59 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451879, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:59.007236 624 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:37:59.008837 624 webserver.cc:460] Webserver started at http://127.3.204.65:46639/ using document root <none> and password file <none>
I20260811 12:37:59.009541 624 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:59.009647 624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:59.015117 624 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 12:37:59.018154 714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:59.019629 624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:37:59.019790 624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "af5fe9d1f56244f990d8a5e608b13fdf"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "be2d5fa4ccdc6d947021af99aabea016"
server_key_iv: "311f9914176b63f025eabb9a7e4ddb99"
server_key_version: "encryptionkey@0"
I20260811 12:37:59.020825 624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:59.038775 624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:59.043183 624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:59.043464 624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:59.044620 624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:37:59.046512 729 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260811 12:37:59.052604 624 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 12:37:59.052759 624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260811 12:37:59.052902 624 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 12:37:59.056353 624 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 12:37:59.056442 624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20260811 12:37:59.056855 729 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Bootstrap starting.
I20260811 12:37:59.085103 729 log.cc:824] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Log is configured to *not* fsync() on all Append() calls
I20260811 12:37:59.093876 624 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:46501
I20260811 12:37:59.093915 911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:46501 every 8 connection(s)
I20260811 12:37:59.095283 624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:37:59.098645 729 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 12:37:59.099236 729 tablet_bootstrap.cc:492] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Bootstrap complete.
I20260811 12:37:59.100543 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:59.098243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:43357)
0811 12:37:59.099173 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:59.099512 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":415,"threads_started":1}
I20260811 12:37:59.100646 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 624
I20260811 12:37:59.100876 729 ts_tablet_manager.cc:1397] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Time spent bootstrapping tablet: real 0.044s user 0.024s sys 0.019s
W20260811 12:37:59.101238 914 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:33833 (0 consecutive failures): Network error: Failed to ping master at 127.3.204.126:33833: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
I20260811 12:37:59.101456 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:42875
--local_ip_for_outbound_sockets=127.3.204.66
--tserver_master_addrs=127.3.204.126:33833
--webserver_port=44809
--webserver_interface=127.3.204.66
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:37:59.102190 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:59.101846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:33967)
0811 12:37:59.102003 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:59.102075 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 12:37:59.103137 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:59.102761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:52501)
0811 12:37:59.102911 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:59.103000 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 12:37:59.103461 914 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 12:37:59.109894 729 raft_consensus.cc:348] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:59.111136 729 raft_consensus.cc:729] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5fe9d1f56244f990d8a5e608b13fdf, State: Initialized, Role: FOLLOWER
I20260811 12:37:59.111933 729 consensus_queue.cc:260] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:37:59.113852 729 ts_tablet_manager.cc:1428] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260811 12:37:59.114053 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:59.113579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:42517)
0811 12:37:59.113782 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:59.113873 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 12:37:59.115078 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:59.114688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:48855)
0811 12:37:59.114854 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:59.114926 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260811 12:37:59.453101 923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:59.453426 923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:59.453579 923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:59.461175 923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:59.461329 923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:59.461508 923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:37:59.471838 923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:42875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=44809
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:59.473388 923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:59.474877 923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:59.489037 932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:59.489055 936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:59.488997 933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:59.489717 923 server_base.cc:1034] running on GCE node
I20260811 12:37:59.490352 923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:59.491325 923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:59.492533 923 hybrid_clock.cc:648] HybridClock initialized: now 1786451879492485 us; error 36 us; skew 500 ppm
I20260811 12:37:59.492848 923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:59 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451879, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:59.496063 923 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:37:59.497576 923 webserver.cc:460] Webserver started at http://127.3.204.66:44809/ using document root <none> and password file <none>
I20260811 12:37:59.498315 923 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:59.498436 923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:59.504523 923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 12:37:59.507431 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:59.508904 923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:37:59.509078 923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "d6b8bb3c552f4efa82899e5ef437a2b0"
format_stamp: "Formatted at 2026-08-11 12:37:13 on dist-test-slave-lgr5"
server_key: "dd7960f815a9b1786c4d11fe20c84bb4"
server_key_iv: "09b5c73d4fb89773416b5e92b1594f95"
server_key_version: "encryptionkey@0"
I20260811 12:37:59.510097 923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:59.522835 923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:59.527228 923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:59.527529 923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:59.528649 923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:37:59.530087 923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:37:59.530181 923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:59.530287 923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:37:59.530357 923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:59.564638 923 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:42875
I20260811 12:37:59.564713 1169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:42875 every 8 connection(s)
I20260811 12:37:59.566239 923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:37:59.571270 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:59.569315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:54945)
0811 12:37:59.570107 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:59.570350 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":325,"threads_started":1}
W20260811 12:37:59.571987 1170 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:33833 (0 consecutive failures): Network error: Failed to ping master at 127.3.204.126:33833: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
I20260811 12:37:59.572841 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:59.572527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:52533)
0811 12:37:59.572659 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:59.572730 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260811 12:37:59.573597 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:37:59.573308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:39465)
0811 12:37:59.573421 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 12:37:59.573494 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 12:37:59.573804 1170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 12:37:59.574683 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 923
I20260811 12:37:59.575344 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:43673
--local_ip_for_outbound_sockets=127.3.204.67
--tserver_master_addrs=127.3.204.126:33833
--webserver_port=41053
--webserver_interface=127.3.204.67
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:37:59.922034 1174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:37:59.922351 1174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:37:59.922451 1174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:37:59.930016 1174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:37:59.930143 1174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:37:59.930263 1174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:37:59.940441 1174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:43673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=41053
--tserver_master_addrs=127.3.204.126:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:37:59.941741 1174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:37:59.943053 1174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:37:59.956665 1184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:59.956633 1183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:37:59.956703 1187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:37:59.957960 1174 server_base.cc:1034] running on GCE node
I20260811 12:37:59.958539 1174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:37:59.959507 1174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:37:59.960705 1174 hybrid_clock.cc:648] HybridClock initialized: now 1786451879960657 us; error 50 us; skew 500 ppm
I20260811 12:37:59.961040 1174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:37:59 dist-test-slave-lgr5 krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451879, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:37:59.964339 1174 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:37:59.965829 1174 webserver.cc:460] Webserver started at http://127.3.204.67:41053/ using document root <none> and password file <none>
I20260811 12:37:59.966545 1174 fs_manager.cc:359] Metadata directory not provided
I20260811 12:37:59.966673 1174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:37:59.972597 1174 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 12:37:59.976634 1195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:37:59.978379 1174 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 12:37:59.978529 1174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "77c04c7984444e42a4db7a19ab042987"
format_stamp: "Formatted at 2026-08-11 12:37:14 on dist-test-slave-lgr5"
server_key: "99e24c6be1313120843dce135a5b2699"
server_key_iv: "0135bcdc61fd18efc3be29f06b9a222f"
server_key_version: "encryptionkey@0"
I20260811 12:37:59.979812 1174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:37:59.993801 1174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:37:59.998266 1174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:37:59.998541 1174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:37:59.999585 1174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:00.001253 1174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:00.001369 1174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:00.001479 1174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:00.001550 1174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:00.033798 1174 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:43673
I20260811 12:38:00.033896 1405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:43673 every 8 connection(s)
I20260811 12:38:00.035344 1174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:38:00.036363 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 1174
I20260811 12:38:00.040771 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.038788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:45537)
0811 12:38:00.039585 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:00.039828 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":289,"threads_started":1}
W20260811 12:38:00.041419 1406 heartbeater.cc:644] Failed to heartbeat to 127.3.204.126:33833 (0 consecutive failures): Network error: Failed to ping master at 127.3.204.126:33833: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
I20260811 12:38:00.042250 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.041890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:56959)
0811 12:38:00.042050 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:00.042123 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 12:38:00.043128 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.042793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:41531)
0811 12:38:00.042942 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:00.043018 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 12:38:00.043373 1406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 12:38:00.117098 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.116070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:43501)
0811 12:38:00.116688 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:00.116795 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":270,"threads_started":1}
I20260811 12:38:00.302065 1414 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:00.302366 1414 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:38:00.304107 1414 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
Aug 11 12:38:00 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451879, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
Aug 11 12:38:00 dist-test-slave-lgr5 krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451879, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:38:00.329659 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.304570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43673 (local address 127.3.204.65:42317)
0811 12:38:00.304734 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:00.304777 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:00.305470 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:00.310255 (+ 4785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:00.310268 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:00.311040 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:00.311833 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:00.311858 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:00.314324 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:00.314328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:00.315982 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:00.315997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:00.317452 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:00.317457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:00.317644 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:00.318412 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:00.318436 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:00.320938 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:00.325310 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:00.325321 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:00.325337 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:00.325837 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:00.326336 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:00.326341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:00.326344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:00.326554 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:00.327791 (+ 1237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:00.327798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:00.328206 (+ 408us) client_negotiation.cc:769] Sending connection context
0811 12:38:00.328559 (+ 353us) client_negotiation.cc:240] Negotiation successful
0811 12:38:00.329223 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":307,"spinlock_wait_cycles":6528}
I20260811 12:38:00.329720 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.304772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:60573 (local address 127.3.204.66:42875)
0811 12:38:00.306929 (+ 2157us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:00.306942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:00.306978 (+ 36us) server_negotiation.cc:407] Connection header received
0811 12:38:00.309044 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:00.309057 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:00.309624 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:00.310023 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:00.312121 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:00.314106 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:00.316161 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:00.317273 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:00.317466 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:00.321588 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:00.321598 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:00.321615 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:00.321664 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:00.325147 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:00.326493 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:00.326500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:00.326506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:00.326719 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:00.327140 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:00.327146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:00.327149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:00.327815 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:00.327977 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:00.328835 (+ 858us) server_negotiation.cc:299] Negotiation successful
0811 12:38:00.329290 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1981,"thread_start_us":1843,"threads_started":1}
I20260811 12:38:00.329711 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.304835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:42317 (local address 127.3.204.67:43673)
0811 12:38:00.306684 (+ 1849us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:00.306703 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:00.306748 (+ 45us) server_negotiation.cc:407] Connection header received
0811 12:38:00.309044 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:00.309057 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:00.309624 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:00.310023 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:00.312071 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:00.314106 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:00.316250 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:00.317262 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:00.317466 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:00.321330 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:00.321344 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:00.321366 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:00.321419 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:00.325162 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:00.325989 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:00.325997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:00.326003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:00.326187 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:00.326735 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:00.326740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:00.326743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:00.327615 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:00.327855 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:00.328845 (+ 990us) server_negotiation.cc:299] Negotiation successful
0811 12:38:00.329279 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":1451,"threads_started":1}
I20260811 12:38:00.329658 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.304570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:42875 (local address 127.3.204.65:60573)
0811 12:38:00.306292 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:00.306312 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:00.306432 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:00.310628 (+ 4196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:00.310632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:00.311119 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:00.311833 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:00.311858 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:00.314356 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:00.314360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:00.315981 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:00.315997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:00.317818 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:00.317823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:00.317956 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:00.318501 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:00.318519 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:00.321255 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:00.325971 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:00.325974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:00.325976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:00.326344 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:00.326875 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:00.326879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:00.326881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:00.327026 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:00.327964 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:00.327968 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:00.328206 (+ 238us) client_negotiation.cc:769] Sending connection context
0811 12:38:00.328559 (+ 353us) client_negotiation.cc:240] Negotiation successful
0811 12:38:00.329083 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1580,"mutex_wait_us":436,"spinlock_wait_cycles":38784,"thread_start_us":1123,"threads_started":1}
W20260811 12:38:00.333438 723 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
W20260811 12:38:00.333712 718 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
I20260811 12:38:00.333864 718 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af5fe9d1f56244f990d8a5e608b13fdf; no voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0
I20260811 12:38:00.334299 1414 raft_consensus.cc:2726] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 12:38:00.575238 1418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:00.574394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:58557)
0811 12:38:00.574851 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:00.575011 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":208,"threads_started":1}
I20260811 12:38:01.044730 1419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:01.043915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:33579)
0811 12:38:01.044403 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:01.044527 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":242,"threads_started":1}
I20260811 12:38:01.119047 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:01.118216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:35249)
0811 12:38:01.118766 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:01.118874 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":313,"threads_started":1}
I20260811 12:38:01.577008 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:01.576271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:42643)
0811 12:38:01.576760 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:01.576848 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260811 12:38:02.046437 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:02.045688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:34009)
0811 12:38:02.046172 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:02.046251 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
I20260811 12:38:02.120968 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:02.120015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:47831)
0811 12:38:02.120655 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:02.120763 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":276,"threads_started":1}
I20260811 12:38:02.326530 1425 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:02.326745 1425 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:38:02.327440 1425 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
W20260811 12:38:02.328574 723 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
W20260811 12:38:02.328747 718 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
I20260811 12:38:02.328869 718 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af5fe9d1f56244f990d8a5e608b13fdf; no voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0
I20260811 12:38:02.329108 1425 raft_consensus.cc:2726] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 12:38:02.578855 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:02.577999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:53977)
0811 12:38:02.578574 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:02.578664 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":316,"threads_started":1}
I20260811 12:38:03.048156 1427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:03.047398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:52653)
0811 12:38:03.047856 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:03.047956 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260811 12:38:03.122629 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:03.121918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:34819)
0811 12:38:03.122373 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:03.122459 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260811 12:38:03.580765 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:03.579910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:55459)
0811 12:38:03.580472 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:03.580567 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":281,"threads_started":1}
I20260811 12:38:04.049937 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:04.049167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:60837)
0811 12:38:04.049637 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:04.049731 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":232,"threads_started":1}
I20260811 12:38:04.124677 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:04.123847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:34183)
0811 12:38:04.124395 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:04.124482 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":298,"threads_started":1}
I20260811 12:38:04.582613 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:04.581824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:41375)
0811 12:38:04.582324 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:04.582430 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260811 12:38:05.051796 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:05.050979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:60603)
0811 12:38:05.051532 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:05.051630 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":305,"threads_started":1}
I20260811 12:38:05.126533 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:05.125669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:48137)
0811 12:38:05.126225 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:05.126363 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":298,"threads_started":1}
I20260811 12:38:05.584395 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:05.583554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:43983)
0811 12:38:05.584108 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:05.584207 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":284,"threads_started":1}
I20260811 12:38:05.615521 1447 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:05.615724 1447 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:38:05.616487 1447 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
W20260811 12:38:05.617391 718 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
W20260811 12:38:05.617574 723 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
I20260811 12:38:05.617669 723 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af5fe9d1f56244f990d8a5e608b13fdf; no voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0
I20260811 12:38:05.617909 1447 raft_consensus.cc:2726] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 12:38:06.053548 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:06.052796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:43695)
0811 12:38:06.053254 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:06.053340 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260811 12:38:06.128626 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:06.127652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:39705)
0811 12:38:06.128271 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:06.128380 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":321,"threads_started":1}
I20260811 12:38:06.586269 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:06.585449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:43277)
0811 12:38:06.585993 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:06.586083 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":225,"threads_started":1}
I20260811 12:38:07.055514 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:07.054590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:33849)
0811 12:38:07.055171 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:07.055293 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":273,"threads_started":1}
I20260811 12:38:07.130584 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:07.129723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:56419)
0811 12:38:07.130313 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:07.130415 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":294,"threads_started":1}
I20260811 12:38:07.588116 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:07.587382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:44067)
0811 12:38:07.587806 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:07.587896 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":210,"threads_started":1}
I20260811 12:38:07.680696 1456 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:07.680919 1456 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:38:07.681705 1456 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
W20260811 12:38:07.682751 718 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
W20260811 12:38:07.683075 723 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
I20260811 12:38:07.683199 723 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af5fe9d1f56244f990d8a5e608b13fdf; no voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0
I20260811 12:38:07.683452 1456 raft_consensus.cc:2726] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 12:38:08.057263 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:08.056509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:38767)
0811 12:38:08.056963 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:08.057062 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":240,"threads_started":1}
I20260811 12:38:08.132654 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:08.131701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:47659)
0811 12:38:08.132307 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:08.132431 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":273,"threads_started":1}
I20260811 12:38:08.589860 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:08.589043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:37027)
0811 12:38:08.589588 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:08.589687 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":260,"threads_started":1}
I20260811 12:38:09.059356 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:09.058451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:45179)
0811 12:38:09.059065 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:09.059183 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":265,"threads_started":1}
I20260811 12:38:09.134579 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:09.133826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:59543)
0811 12:38:09.134316 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:09.134410 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260811 12:38:09.347342 1464 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:09.347540 1464 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:38:09.348275 1464 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
W20260811 12:38:09.349248 723 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
W20260811 12:38:09.349440 718 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
I20260811 12:38:09.349543 718 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af5fe9d1f56244f990d8a5e608b13fdf; no voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0
I20260811 12:38:09.349762 1464 raft_consensus.cc:2726] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 12:38:09.591735 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:09.590829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:52437)
0811 12:38:09.591375 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:09.591487 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":276,"threads_started":1}
I20260811 12:38:10.061146 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:10.060404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:36759)
0811 12:38:10.060867 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:10.060964 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260811 12:38:10.136433 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:10.135606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:44093)
0811 12:38:10.136136 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:10.136231 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260811 12:38:10.593638 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:10.592801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:44035)
0811 12:38:10.593363 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:10.593469 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":265,"threads_started":1}
I20260811 12:38:11.062973 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:11.062117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:40493)
0811 12:38:11.062686 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:11.062798 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":286,"threads_started":1}
I20260811 12:38:11.118122 1480 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:11.118346 1480 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:38:11.118983 1480 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
W20260811 12:38:11.120031 718 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
W20260811 12:38:11.120223 723 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
I20260811 12:38:11.120337 723 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af5fe9d1f56244f990d8a5e608b13fdf; no voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0
I20260811 12:38:11.120554 1480 raft_consensus.cc:2726] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 12:38:11.138314 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:11.137449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:47779)
0811 12:38:11.137963 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:11.138100 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":239,"threads_started":1}
I20260811 12:38:11.595580 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:11.594762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:57481)
0811 12:38:11.595296 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:11.595399 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":282,"threads_started":1}
I20260811 12:38:12.064914 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:12.064117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:44005)
0811 12:38:12.064654 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:12.064734 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":310,"threads_started":1}
I20260811 12:38:12.140303 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:12.139475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:40553)
0811 12:38:12.139972 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:12.140069 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":244,"threads_started":1}
I20260811 12:38:12.597374 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:12.596602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:53241)
0811 12:38:12.597104 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:12.597198 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
I20260811 12:38:13.066721 1500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:13.065946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:47639)
0811 12:38:13.066452 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:13.066551 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":245,"threads_started":1}
I20260811 12:38:13.142163 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:13.141291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:56721)
0811 12:38:13.141832 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:13.141941 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":263,"threads_started":1}
I20260811 12:38:13.596382 1503 raft_consensus.cc:482] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:13.596762 1503 raft_consensus.cc:504] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c04c7984444e42a4db7a19ab042987" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43673 } } peers { permanent_uuid: "d6b8bb3c552f4efa82899e5ef437a2b0" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 42875 } } peers { permanent_uuid: "af5fe9d1f56244f990d8a5e608b13fdf" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46501 } }
I20260811 12:38:13.597481 1503 leader_election.cc:288] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673), d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875)
W20260811 12:38:13.598670 718 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77c04c7984444e42a4db7a19ab042987 (127.3.204.67:43673): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
W20260811 12:38:13.598861 723 leader_election.cc:341] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b8bb3c552f4efa82899e5ef437a2b0 (127.3.204.66:42875): Not found: Tablet not found: 818ff82eeff74fd7898eea6bd2af6573
I20260811 12:38:13.598995 723 leader_election.cc:302] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af5fe9d1f56244f990d8a5e608b13fdf; no voters: 77c04c7984444e42a4db7a19ab042987, d6b8bb3c552f4efa82899e5ef437a2b0
I20260811 12:38:13.599252 1503 raft_consensus.cc:2726] T 818ff82eeff74fd7898eea6bd2af6573 P af5fe9d1f56244f990d8a5e608b13fdf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 12:38:13.599500 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:13.598279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:38791)
0811 12:38:13.599151 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:13.599265 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":621,"threads_started":1}
I20260811 12:38:14.068634 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:14.067781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:55609)
0811 12:38:14.068307 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:14.068399 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":288,"threads_started":1}
I20260811 12:38:14.144335 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:14.143502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:33609)
0811 12:38:14.143985 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:14.144135 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":246,"threads_started":1}
I20260811 12:38:14.601506 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:14.600803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.66:42979)
0811 12:38:14.601235 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:14.601343 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260811 12:38:15.040552 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 624
I20260811 12:38:15.055929 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:15.041711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.3.204.65:46501)
0811 12:38:15.042269 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:15.042279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:15.042314 (+ 35us) server_negotiation.cc:407] Connection header received
0811 12:38:15.042409 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:15.042418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:15.042722 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:15.042884 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:15.044457 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:15.045447 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:15.046736 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:15.047315 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:15.047440 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:15.050155 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:15.050167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:15.050182 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:15.050217 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:15.052999 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:15.053735 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:15.053743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:15.053746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:15.053886 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:15.054257 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:15.054261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:15.054263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:15.054662 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:15.054968 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:15.055302 (+ 334us) server_negotiation.cc:299] Negotiation successful
0811 12:38:15.055610 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":634,"threads_started":1}
I20260811 12:38:15.057399 906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:15.070762 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:15.069974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.67:58801)
0811 12:38:15.070461 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:15.070590 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260811 12:38:15.149257 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:15.147769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33833 (local address 127.3.204.65:49197)
0811 12:38:15.148763 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:15.148947 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.126:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":332,"threads_started":1}
I20260811 12:38:15.152544 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 624
I20260811 12:38:15.176581 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 923
I20260811 12:38:15.190480 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:15.177327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.3.204.66:42875)
0811 12:38:15.177812 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:15.177817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:15.177833 (+ 16us) server_negotiation.cc:407] Connection header received
0811 12:38:15.177891 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:15.177895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:15.177954 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:15.178046 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:15.179548 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:15.180403 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:15.181681 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:15.182137 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:15.182234 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:15.185178 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:15.185184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:15.185188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:15.185223 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:15.187858 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:15.188691 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:15.188696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:15.188698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:15.188833 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:15.189238 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:15.189241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:15.189243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:15.189527 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:15.189726 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:15.189989 (+ 263us) server_negotiation.cc:299] Negotiation successful
0811 12:38:15.190167 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":270,"threads_started":1}
I20260811 12:38:15.191195 1161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:15.274607 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 923
I20260811 12:38:15.295338 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 1174
I20260811 12:38:15.309321 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:15.296172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.3.204.67:43673)
0811 12:38:15.296729 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:15.296734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:15.296752 (+ 18us) server_negotiation.cc:407] Connection header received
0811 12:38:15.296812 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:15.296815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:15.296885 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:15.296986 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:15.298364 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:15.299178 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:15.300485 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:15.300938 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:15.301038 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:15.303692 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:15.303698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:15.303701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:15.303742 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:15.306569 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:15.307340 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:15.307344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:15.307347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:15.307526 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:15.307940 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:15.307945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:15.307948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:15.308336 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:15.308558 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:15.308815 (+ 257us) server_negotiation.cc:299] Negotiation successful
0811 12:38:15.309015 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":293,"threads_started":1}
I20260811 12:38:15.310088 1395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:15.391908 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 1174
2026-08-11T12:38:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64988 ms)
[----------] 30 tests from SecurityITest (310053 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:15 dist-test-slave-lgr5 krb5kdc[1529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:15 dist-test-slave-lgr5 krb5kdc[1529](info): set up 2 sockets
Aug 11 12:38:15 dist-test-slave-lgr5 krb5kdc[1529](info): commencing operation
krb5kdc: starting...
W20260811 12:38:16.682919 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:16 dist-test-slave-lgr5 krb5kdc[1529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:38:16.997140 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:39481
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:39481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:17.348497 1549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:17.348803 1549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:17.348940 1549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:17.356293 1549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:17.356413 1549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:17.356489 1549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:17.356566 1549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:17.356626 1549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:17.366801 1549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:39481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:39481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:17.368194 1549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:17.369526 1549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:17.381704 1561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:17.381676 1568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:17.381825 1549 server_base.cc:1034] running on GCE node
W20260811 12:38:17.381896 1562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:17.383133 1549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:17.384728 1549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:17.385946 1549 hybrid_clock.cc:648] HybridClock initialized: now 1786451897385879 us; error 54 us; skew 500 ppm
I20260811 12:38:17.386298 1549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:17 dist-test-slave-lgr5 krb5kdc[1529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451897, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:17.392529 1549 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:38:17.394109 1549 webserver.cc:460] Webserver started at http://127.3.204.126:39697/ using document root <none> and password file <none>
I20260811 12:38:17.394855 1549 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:17.394996 1549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:17.395298 1549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:17.398494 1549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "eaaf516a534e476c8e00812edaa168c0"
format_stamp: "Formatted at 2026-08-11 12:38:17 on dist-test-slave-lgr5"
server_key: "1317d9b29aa8c7f2335b3d649fb9779e"
server_key_iv: "c4c8ec69894740f0472d39675faa34ed"
server_key_version: "encryptionkey@0"
I20260811 12:38:17.399292 1549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "eaaf516a534e476c8e00812edaa168c0"
format_stamp: "Formatted at 2026-08-11 12:38:17 on dist-test-slave-lgr5"
server_key: "1317d9b29aa8c7f2335b3d649fb9779e"
server_key_iv: "c4c8ec69894740f0472d39675faa34ed"
server_key_version: "encryptionkey@0"
I20260811 12:38:17.404804 1549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260811 12:38:17.408591 1583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:17.410015 1549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 12:38:17.410190 1549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "eaaf516a534e476c8e00812edaa168c0"
format_stamp: "Formatted at 2026-08-11 12:38:17 on dist-test-slave-lgr5"
server_key: "1317d9b29aa8c7f2335b3d649fb9779e"
server_key_iv: "c4c8ec69894740f0472d39675faa34ed"
server_key_version: "encryptionkey@0"
I20260811 12:38:17.410384 1549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:17.421653 1549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:17.425881 1549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:17.426177 1549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:17.443743 1549 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:39481
I20260811 12:38:17.443765 1680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:39481 every 8 connection(s)
I20260811 12:38:17.445228 1549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:17.447986 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 1549
I20260811 12:38:17.448266 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:17.449100 3889 external_mini_cluster.cc:1428] Setting key 393df398b082edd81971174eb5935db4
I20260811 12:38:17.449743 1682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:17.464350 1682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0: Bootstrap starting.
Aug 11 12:38:17 dist-test-slave-lgr5 krb5kdc[1529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:17.467952 1682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:17.469136 1682 log.cc:824] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:17.472680 1682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0: No bootstrap required, opened a new log
I20260811 12:38:17.474653 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:17.451175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.3.204.126:39481)
0811 12:38:17.451976 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:17.451991 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:17.452042 (+ 51us) server_negotiation.cc:407] Connection header received
0811 12:38:17.454619 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:17.454653 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:17.455229 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:17.455717 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:17.457174 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:17.459155 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:17.460807 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:17.461920 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:17.462133 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:17.466694 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:17.466703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:17.466719 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:17.466764 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:17.470113 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:17.470957 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:17.470966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:17.470974 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:17.471191 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:17.471626 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:17.471631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:17.471634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:17.472413 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:17.472725 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:17.473177 (+ 452us) server_negotiation.cc:299] Negotiation successful
0811 12:38:17.473630 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":321,"threads_started":1}
I20260811 12:38:17.481364 1682 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaf516a534e476c8e00812edaa168c0" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39481 } }
I20260811 12:38:17.481755 1682 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:17.481856 1682 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf516a534e476c8e00812edaa168c0, State: Initialized, Role: FOLLOWER
I20260811 12:38:17.482393 1682 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaf516a534e476c8e00812edaa168c0" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39481 } }
I20260811 12:38:17.482616 1682 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:17.482740 1682 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:17.482905 1682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:17.485185 1682 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaf516a534e476c8e00812edaa168c0" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39481 } }
I20260811 12:38:17.485708 1682 leader_election.cc:302] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaaf516a534e476c8e00812edaa168c0; no voters:
I20260811 12:38:17.486424 1682 leader_election.cc:288] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:17.486574 1691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:17.487766 1691 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [term 1 LEADER]: Becoming Leader. State: Replica: eaaf516a534e476c8e00812edaa168c0, State: Running, Role: LEADER
I20260811 12:38:17.488358 1691 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaf516a534e476c8e00812edaa168c0" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39481 } }
I20260811 12:38:17.488862 1682 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:17.491029 1695 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaaf516a534e476c8e00812edaa168c0. Latest consensus state: current_term: 1 leader_uuid: "eaaf516a534e476c8e00812edaa168c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaf516a534e476c8e00812edaa168c0" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39481 } } }
I20260811 12:38:17.491532 1695 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:17.491447 1692 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaaf516a534e476c8e00812edaa168c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaf516a534e476c8e00812edaa168c0" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39481 } } }
I20260811 12:38:17.491737 1692 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:17.493268 1708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:17.500267 1708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:17.510069 1708 catalog_manager.cc:1347] Generated new cluster ID: 10a96e1d6c1c4896a3888510230ea1d6
I20260811 12:38:17.510192 1708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:17.523043 1708 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:17.524412 1708 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:17.532006 1708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0: Generated new TSK 0
I20260811 12:38:17.532927 1708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:38:17.615887 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39481
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:17.963789 1732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:17.964144 1732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:17.964293 1732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:17.971911 1732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:17.972138 1732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:17.972348 1732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:38:17.986405 1732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:17.987928 1732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:17.989447 1732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:18.003391 1752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:18.003283 1750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:18.003299 1746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:18.004230 1732 server_base.cc:1034] running on GCE node
I20260811 12:38:18.004935 1732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:18.005941 1732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:18.007174 1732 hybrid_clock.cc:648] HybridClock initialized: now 1786451898007109 us; error 49 us; skew 500 ppm
I20260811 12:38:18.007515 1732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:18 dist-test-slave-lgr5 krb5kdc[1529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:18.010680 1732 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:38:18.012257 1732 webserver.cc:460] Webserver started at http://127.3.204.65:41989/ using document root <none> and password file <none>
I20260811 12:38:18.012974 1732 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:18.013111 1732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:18.013422 1732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:18.016505 1732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "0d52217431c3435e8d0adbd1688a61af"
format_stamp: "Formatted at 2026-08-11 12:38:18 on dist-test-slave-lgr5"
server_key: "674c8c40578527385ad44a69b4d1c4c5"
server_key_iv: "ee723573eb727d9a8e670eaef71f8efe"
server_key_version: "encryptionkey@0"
I20260811 12:38:18.017315 1732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "0d52217431c3435e8d0adbd1688a61af"
format_stamp: "Formatted at 2026-08-11 12:38:18 on dist-test-slave-lgr5"
server_key: "674c8c40578527385ad44a69b4d1c4c5"
server_key_iv: "ee723573eb727d9a8e670eaef71f8efe"
server_key_version: "encryptionkey@0"
I20260811 12:38:18.022734 1732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 12:38:18.026475 1767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:18.027866 1732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:38:18.028052 1732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "0d52217431c3435e8d0adbd1688a61af"
format_stamp: "Formatted at 2026-08-11 12:38:18 on dist-test-slave-lgr5"
server_key: "674c8c40578527385ad44a69b4d1c4c5"
server_key_iv: "ee723573eb727d9a8e670eaef71f8efe"
server_key_version: "encryptionkey@0"
I20260811 12:38:18.028308 1732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:18.050418 1732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:18.054840 1732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:18.055156 1732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:18.056252 1732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:18.057693 1732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:18.057778 1732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:18.057904 1732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:18.057957 1732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:18.090338 1732 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:33017
I20260811 12:38:18.090443 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:33017 every 8 connection(s)
I20260811 12:38:18.091696 1732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:38:18.099217 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 1732
I20260811 12:38:18.099409 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:38:18.100101 3889 external_mini_cluster.cc:1428] Setting key 4d66a66a7daf0d1270fe60439efbeeef
Aug 11 12:38:18 dist-test-slave-lgr5 krb5kdc[1529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:18.113082 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:18.094383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:52099 (local address 127.3.204.126:39481)
0811 12:38:18.094835 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:18.094840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:18.095849 (+ 1009us) server_negotiation.cc:407] Connection header received
0811 12:38:18.097767 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:18.097773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:18.097857 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:18.097957 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:18.099605 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:18.100439 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:18.102654 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:18.103072 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:18.103193 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:18.106906 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:18.106911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:18.106915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:18.106950 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:18.109772 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:18.110592 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:18.110597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:18.110599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:18.110727 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:18.111205 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:18.111209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:18.111211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:18.111503 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:18.111655 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:18.112486 (+ 831us) server_negotiation.cc:299] Negotiation successful
0811 12:38:18.112728 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260811 12:38:18.114102 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:18.094956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:39481 (local address 127.3.204.65:52099)
0811 12:38:18.095672 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:18.095715 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:18.097478 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:18.098229 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:18.098237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:18.098759 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:18.099384 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:18.099407 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:18.100621 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:18.100627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:18.102499 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:18.102514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:18.103324 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:18.103332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:18.103595 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:18.104363 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:18.104393 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:18.106652 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:18.109942 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:18.109953 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:18.109971 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:18.110463 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:18.110841 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:18.110847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:18.110851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:18.111086 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:18.111636 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:18.111644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:18.112028 (+ 384us) client_negotiation.cc:769] Sending connection context
0811 12:38:18.112357 (+ 329us) client_negotiation.cc:240] Negotiation successful
0811 12:38:18.112956 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":258,"threads_started":1}
I20260811 12:38:18.115692 1988 heartbeater.cc:347] Connected to a master server at 127.3.204.126:39481
I20260811 12:38:18.116000 1988 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:18.116746 1988 heartbeater.cc:511] Master 127.3.204.126:39481 requested a full tablet report, sending...
I20260811 12:38:18.118813 1613 ts_manager.cc:194] Registered new tserver with Master: 0d52217431c3435e8d0adbd1688a61af (127.3.204.65:33017)
I20260811 12:38:18.120390 1613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:52099
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:38:18.167515 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39481
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:18.514549 1998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:18.514859 1998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:18.514955 1998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:18.522379 1998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:18.522507 1998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:18.522620 1998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:38:18.532866 1998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:18.534209 1998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:18.535570 1998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:18.549487 2008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:18.549479 2015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:18.549567 2010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:18.550382 1998 server_base.cc:1034] running on GCE node
I20260811 12:38:18.551033 1998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:18.552048 1998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:18.553329 1998 hybrid_clock.cc:648] HybridClock initialized: now 1786451898553258 us; error 51 us; skew 500 ppm
I20260811 12:38:18.553673 1998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:18 dist-test-slave-lgr5 krb5kdc[1529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:18.557018 1998 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:38:18.558703 1998 webserver.cc:460] Webserver started at http://127.3.204.66:36679/ using document root <none> and password file <none>
I20260811 12:38:18.559478 1998 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:18.559609 1998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:18.559952 1998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:18.563089 1998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "4af4b4a9d34046008efc05e88bd8eedb"
format_stamp: "Formatted at 2026-08-11 12:38:18 on dist-test-slave-lgr5"
server_key: "8b2b00a7a583fe4ef6d57df8da9442af"
server_key_iv: "9d5cfe58431dafcc56d49fcaf1e1d4e2"
server_key_version: "encryptionkey@0"
I20260811 12:38:18.563923 1998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "4af4b4a9d34046008efc05e88bd8eedb"
format_stamp: "Formatted at 2026-08-11 12:38:18 on dist-test-slave-lgr5"
server_key: "8b2b00a7a583fe4ef6d57df8da9442af"
server_key_iv: "9d5cfe58431dafcc56d49fcaf1e1d4e2"
server_key_version: "encryptionkey@0"
I20260811 12:38:18.569469 1998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 12:38:18.573559 2028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:18.575049 1998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:18.575229 1998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "4af4b4a9d34046008efc05e88bd8eedb"
format_stamp: "Formatted at 2026-08-11 12:38:18 on dist-test-slave-lgr5"
server_key: "8b2b00a7a583fe4ef6d57df8da9442af"
server_key_iv: "9d5cfe58431dafcc56d49fcaf1e1d4e2"
server_key_version: "encryptionkey@0"
I20260811 12:38:18.575453 1998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:18.599037 1998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:18.603551 1998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:18.603848 1998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:18.604971 1998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:18.606359 1998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:18.606446 1998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:18.606575 1998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:18.606629 1998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:18.638337 1998 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:36483
I20260811 12:38:18.638409 2234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:36483 every 8 connection(s)
I20260811 12:38:18.639765 1998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:38:18.649215 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 1998
I20260811 12:38:18.649376 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:38:18.649807 3889 external_mini_cluster.cc:1428] Setting key a1012a8d8fa9d464dcff57d2f0be6885
Aug 11 12:38:18 dist-test-slave-lgr5 krb5kdc[1529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:18.661690 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:18.642417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:33997 (local address 127.3.204.126:39481)
0811 12:38:18.642910 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:18.642917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:18.643935 (+ 1018us) server_negotiation.cc:407] Connection header received
0811 12:38:18.646044 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:18.646048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:18.646125 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:18.646216 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:18.647949 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:18.648729 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:18.650545 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:18.651119 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:18.651259 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:18.655109 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:18.655115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:18.655119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:18.655161 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:18.658272 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:18.659173 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:18.659178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:18.659182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:18.659345 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:18.659823 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:18.659828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:18.659830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:18.660220 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:18.660375 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:18.661125 (+ 750us) server_negotiation.cc:299] Negotiation successful
0811 12:38:18.661349 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":228,"threads_started":1}
I20260811 12:38:18.662597 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:18.642909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:39481 (local address 127.3.204.66:33997)
0811 12:38:18.643745 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:18.643804 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:18.645706 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:18.646507 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:18.646515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:18.647082 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:18.647742 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:18.647762 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:18.648903 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:18.648907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:18.650373 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:18.650388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:18.651404 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:18.651411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:18.651634 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:18.652357 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:18.652382 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:18.654816 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:18.658469 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:18.658477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:18.658493 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:18.659038 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:18.659464 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:18.659468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:18.659470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:18.659693 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:18.660360 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:18.660367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:18.660743 (+ 376us) client_negotiation.cc:769] Sending connection context
0811 12:38:18.661000 (+ 257us) client_negotiation.cc:240] Negotiation successful
0811 12:38:18.661432 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":329,"threads_started":1}
I20260811 12:38:18.664149 2235 heartbeater.cc:347] Connected to a master server at 127.3.204.126:39481
I20260811 12:38:18.664503 2235 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:18.665403 2235 heartbeater.cc:511] Master 127.3.204.126:39481 requested a full tablet report, sending...
I20260811 12:38:18.667065 1613 ts_manager.cc:194] Registered new tserver with Master: 4af4b4a9d34046008efc05e88bd8eedb (127.3.204.66:36483)
I20260811 12:38:18.667901 1613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:33997
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:38:18.717592 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39481
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:19.061479 2247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:19.061802 2247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:19.061954 2247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:19.069398 2247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:19.069536 2247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:19.069695 2247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:38:19.080040 2247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:19.081461 2247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:19.082790 2247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:19.096720 2258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:19.096783 2254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:19.096680 2253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:19.105772 2247 server_base.cc:1034] running on GCE node
I20260811 12:38:19.106551 2247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:19.107676 2247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:19.108886 2247 hybrid_clock.cc:648] HybridClock initialized: now 1786451899108830 us; error 43 us; skew 500 ppm
I20260811 12:38:19.109210 2247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:19 dist-test-slave-lgr5 krb5kdc[1529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451899, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:19.112607 2247 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:38:19.114300 2247 webserver.cc:460] Webserver started at http://127.3.204.67:35473/ using document root <none> and password file <none>
I20260811 12:38:19.115094 2247 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:19.115208 2247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:19.115531 2247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:19.118687 2247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9"
format_stamp: "Formatted at 2026-08-11 12:38:19 on dist-test-slave-lgr5"
server_key: "ec952db8209720c7d0f5fa5bfd93fdaa"
server_key_iv: "d3b7a9694756d707fa3985e36c2a867f"
server_key_version: "encryptionkey@0"
I20260811 12:38:19.119484 2247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9"
format_stamp: "Formatted at 2026-08-11 12:38:19 on dist-test-slave-lgr5"
server_key: "ec952db8209720c7d0f5fa5bfd93fdaa"
server_key_iv: "d3b7a9694756d707fa3985e36c2a867f"
server_key_version: "encryptionkey@0"
I20260811 12:38:19.123595 1988 heartbeater.cc:503] Master 127.3.204.126:39481 was elected leader, sending a full tablet report...
I20260811 12:38:19.125097 2247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 12:38:19.128858 2268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:19.130285 2247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:19.130451 2247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9"
format_stamp: "Formatted at 2026-08-11 12:38:19 on dist-test-slave-lgr5"
server_key: "ec952db8209720c7d0f5fa5bfd93fdaa"
server_key_iv: "d3b7a9694756d707fa3985e36c2a867f"
server_key_version: "encryptionkey@0"
I20260811 12:38:19.130666 2247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:19.163710 2247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:19.168030 2247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:19.168345 2247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:19.169353 2247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:19.170768 2247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:19.170842 2247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:19.170972 2247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:19.171023 2247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:19.203496 2247 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:44407
I20260811 12:38:19.203528 2499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:44407 every 8 connection(s)
I20260811 12:38:19.204880 2247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:38:19.208902 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 2247
I20260811 12:38:19.209098 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:38:19.209561 3889 external_mini_cluster.cc:1428] Setting key c6bf07920abd0aedfadfd071d7b9d780
Aug 11 12:38:19 dist-test-slave-lgr5 krb5kdc[1529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451899, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:19.225364 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.207536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:51135 (local address 127.3.204.126:39481)
0811 12:38:19.208010 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.208015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.208953 (+ 938us) server_negotiation.cc:407] Connection header received
0811 12:38:19.211008 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.211014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.211110 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.211204 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:19.212785 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.213564 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.215260 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.215694 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.215805 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.219174 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:19.219179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:19.219183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:19.219217 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:19.221856 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:19.222851 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:19.222855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:19.222858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:19.223007 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:19.223458 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:19.223463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:19.223466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:19.223772 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:19.223921 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.224731 (+ 810us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.224984 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260811 12:38:19.226346 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.208042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:39481 (local address 127.3.204.67:51135)
0811 12:38:19.208796 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:19.208842 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:19.210731 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:19.211456 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:19.211467 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:19.212001 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:19.212621 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.212639 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.213739 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.213743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.215119 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.215134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.215924 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.215931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.216218 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.216821 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:19.216845 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:19.218954 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:19.222046 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:19.222054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:19.222069 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:19.222694 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:19.223120 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:19.223125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:19.223128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:19.223335 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:19.223906 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:19.223914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:19.224327 (+ 413us) client_negotiation.cc:769] Sending connection context
0811 12:38:19.224593 (+ 266us) client_negotiation.cc:240] Negotiation successful
0811 12:38:19.225156 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":300,"threads_started":1}
I20260811 12:38:19.227900 2500 heartbeater.cc:347] Connected to a master server at 127.3.204.126:39481
I20260811 12:38:19.228245 2500 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:19.229012 2500 heartbeater.cc:511] Master 127.3.204.126:39481 requested a full tablet report, sending...
I20260811 12:38:19.230507 1613 ts_manager.cc:194] Registered new tserver with Master: 5d1f555bd3b14e7d9cb63be8aa60fcd9 (127.3.204.67:44407)
I20260811 12:38:19.231315 1613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:51135
I20260811 12:38:19.237138 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:38:19.254318 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.241996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.3.204.126:39481)
0811 12:38:19.242225 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.242233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.242618 (+ 385us) server_negotiation.cc:407] Connection header received
0811 12:38:19.242729 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.242733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.242798 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.242878 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:19.244319 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.245070 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.246346 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.246740 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.246830 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.249219 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:19.249224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:19.249228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:19.249264 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:19.251888 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:19.252596 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:19.252599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:19.252602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:19.252730 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:19.253016 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:19.253018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:19.253020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:19.253349 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:19.253543 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.253819 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.254014 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:38:19.258535 1613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:38:19.265520 1613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:38:19.289558 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.278128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.3.204.66:36483)
0811 12:38:19.278704 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.278713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.279405 (+ 692us) server_negotiation.cc:407] Connection header received
0811 12:38:19.279900 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.279909 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.280228 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.280405 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:19.281416 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.283117 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.286214 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.287815 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.287938 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.288206 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.288970 (+ 764us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.289174 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":301,"threads_started":1}
I20260811 12:38:19.290520 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.277930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:36483 (local address 127.0.0.1:33900)
0811 12:38:19.279256 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:19.279295 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:19.279641 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:19.280769 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:19.280778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:19.280813 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:19.281218 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.281237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.283319 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.283325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.285974 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.285994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.288431 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.288437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.288666 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.288718 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 12:38:19.289869 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0811 12:38:19.290135 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"threads_started":1}
I20260811 12:38:19.296921 2105 tablet_service.cc:1467] Processing CreateTablet for tablet 843201827fc24cd8a28c5961d43548ef (DEFAULT_TABLE table=test-table [id=ab6234f3890f483586223829cab3367f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:19.298436 2105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843201827fc24cd8a28c5961d43548ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:19.299378 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.279330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:44407 (local address 127.0.0.1:60110)
0811 12:38:19.292146 (+ 12816us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:19.292165 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:19.292300 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:19.293166 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:19.293171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:19.293200 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:19.293618 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.293631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.295038 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.295044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.296914 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.296931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.298604 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.298611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.298777 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.298813 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:38:19.298910 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 12:38:19.299041 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12669,"thread_start_us":12497,"threads_started":1}
I20260811 12:38:19.299499 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.279994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.3.204.67:44407)
0811 12:38:19.291666 (+ 11672us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.291676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.292473 (+ 797us) server_negotiation.cc:407] Connection header received
0811 12:38:19.292602 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.292610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.292897 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.293158 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:19.293818 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.294884 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.297115 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.298383 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.298523 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.298749 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.298984 (+ 235us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.299147 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11485,"thread_start_us":11288,"threads_started":1}
I20260811 12:38:19.300660 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.278310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33017 (local address 127.0.0.1:57778)
0811 12:38:19.290838 (+ 12528us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:19.290859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:19.290989 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:19.293544 (+ 2555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:19.293548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:19.293581 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:19.293827 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.293840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.296006 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.296012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.297826 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.297842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.299891 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.299897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.300069 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.300164 (+ 95us) client_negotiation.cc:769] Sending connection context
0811 12:38:19.300256 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 12:38:19.300383 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12392,"thread_start_us":557,"threads_started":1}
I20260811 12:38:19.300804 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.278383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.3.204.65:33017)
0811 12:38:19.292920 (+ 14537us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.292928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.292956 (+ 28us) server_negotiation.cc:407] Connection header received
0811 12:38:19.293056 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.293063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.293299 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.294013 (+ 714us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:19.294287 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.295826 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.298013 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.299652 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.299852 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.300074 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.300309 (+ 235us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.300491 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14358,"thread_start_us":14197,"threads_started":1}
I20260811 12:38:19.303406 2360 tablet_service.cc:1467] Processing CreateTablet for tablet 843201827fc24cd8a28c5961d43548ef (DEFAULT_TABLE table=test-table [id=ab6234f3890f483586223829cab3367f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:19.305114 2360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843201827fc24cd8a28c5961d43548ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:19.304791 1860 tablet_service.cc:1467] Processing CreateTablet for tablet 843201827fc24cd8a28c5961d43548ef (DEFAULT_TABLE table=test-table [id=ab6234f3890f483586223829cab3367f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:19.306228 1860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843201827fc24cd8a28c5961d43548ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:19.311537 2531 tablet_bootstrap.cc:492] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Bootstrap starting.
I20260811 12:38:19.314852 2531 tablet_bootstrap.cc:654] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:19.315944 2531 log.cc:824] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:19.321388 2538 tablet_bootstrap.cc:492] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Bootstrap starting.
I20260811 12:38:19.322042 2537 tablet_bootstrap.cc:492] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Bootstrap starting.
I20260811 12:38:19.322654 2531 tablet_bootstrap.cc:492] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: No bootstrap required, opened a new log
I20260811 12:38:19.322953 2531 ts_tablet_manager.cc:1397] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260811 12:38:19.325090 2537 tablet_bootstrap.cc:654] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:19.325300 2538 tablet_bootstrap.cc:654] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:19.326176 2537 log.cc:824] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:19.326529 2538 log.cc:824] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:19.329308 2538 tablet_bootstrap.cc:492] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: No bootstrap required, opened a new log
I20260811 12:38:19.329463 2537 tablet_bootstrap.cc:492] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: No bootstrap required, opened a new log
I20260811 12:38:19.329684 2538 ts_tablet_manager.cc:1397] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 12:38:19.329723 2537 ts_tablet_manager.cc:1397] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260811 12:38:19.333525 2531 raft_consensus.cc:348] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.334039 2531 raft_consensus.cc:374] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:19.334146 2531 raft_consensus.cc:729] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af4b4a9d34046008efc05e88bd8eedb, State: Initialized, Role: FOLLOWER
I20260811 12:38:19.334828 2531 consensus_queue.cc:260] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.336203 2235 heartbeater.cc:503] Master 127.3.204.126:39481 was elected leader, sending a full tablet report...
I20260811 12:38:19.336897 2531 ts_tablet_manager.cc:1428] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260811 12:38:19.338598 2538 raft_consensus.cc:348] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.339090 2538 raft_consensus.cc:374] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:19.339195 2538 raft_consensus.cc:729] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d52217431c3435e8d0adbd1688a61af, State: Initialized, Role: FOLLOWER
I20260811 12:38:19.339787 2538 consensus_queue.cc:260] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.339917 2537 raft_consensus.cc:348] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.340466 2537 raft_consensus.cc:374] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:19.340615 2537 raft_consensus.cc:729] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1f555bd3b14e7d9cb63be8aa60fcd9, State: Initialized, Role: FOLLOWER
I20260811 12:38:19.341215 2537 consensus_queue.cc:260] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.341657 2538 ts_tablet_manager.cc:1428] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260811 12:38:19.343108 2500 heartbeater.cc:503] Master 127.3.204.126:39481 was elected leader, sending a full tablet report...
I20260811 12:38:19.343328 2537 ts_tablet_manager.cc:1428] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
W20260811 12:38:19.343976 1989 tablet.cc:2369] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:38:19.391872 2236 tablet.cc:2369] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:38:19.456573 2503 tablet.cc:2369] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:38:19.654464 2547 raft_consensus.cc:482] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:19.654742 2547 raft_consensus.cc:504] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.656584 2547 leader_election.cc:288] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d52217431c3435e8d0adbd1688a61af (127.3.204.65:33017), 5d1f555bd3b14e7d9cb63be8aa60fcd9 (127.3.204.67:44407)
I20260811 12:38:19.663311 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.657103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:49557 (local address 127.3.204.65:33017)
0811 12:38:19.657338 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.657344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.658025 (+ 681us) server_negotiation.cc:407] Connection header received
0811 12:38:19.658216 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.658221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.658284 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.658358 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:19.658900 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.659704 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.661578 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.662372 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.662523 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.662605 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.662907 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.663048 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 12:38:19.663322 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.657205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:48775 (local address 127.3.204.67:44407)
0811 12:38:19.657407 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.657413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.658152 (+ 739us) server_negotiation.cc:407] Connection header received
0811 12:38:19.658314 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.658318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.658379 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.658456 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:19.659109 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.659848 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.661620 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.662378 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.662550 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.662626 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.662950 (+ 324us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.663082 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 12:38:19.663491 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.656795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:33017 (local address 127.3.204.66:49557)
0811 12:38:19.657897 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:19.657914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:19.658076 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:19.658385 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:19.658389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:19.658414 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:19.658751 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.658760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.659844 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.659847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.661417 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.661429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.662605 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.662613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.662760 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.662784 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 12:38:19.662867 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 12:38:19.663041 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":42,"thread_start_us":763,"threads_started":1}
I20260811 12:38:19.663537 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.657100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:44407 (local address 127.3.204.66:48775)
0811 12:38:19.658032 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:19.658051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:19.658179 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:19.658485 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:19.658489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:19.658531 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:19.658930 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.658939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.659990 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.659994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.661462 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:19.661476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.662659 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.662665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:19.662800 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.662823 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 12:38:19.662902 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 12:38:19.663052 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":7,"thread_start_us":522,"threads_started":1}
I20260811 12:38:19.664664 2402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843201827fc24cd8a28c5961d43548ef" candidate_uuid: "4af4b4a9d34046008efc05e88bd8eedb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" is_pre_election: true
I20260811 12:38:19.664664 1900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843201827fc24cd8a28c5961d43548ef" candidate_uuid: "4af4b4a9d34046008efc05e88bd8eedb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d52217431c3435e8d0adbd1688a61af" is_pre_election: true
I20260811 12:38:19.665138 2402 raft_consensus.cc:2454] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af4b4a9d34046008efc05e88bd8eedb in term 0.
I20260811 12:38:19.665143 1900 raft_consensus.cc:2454] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af4b4a9d34046008efc05e88bd8eedb in term 0.
I20260811 12:38:19.665748 2030 leader_election.cc:302] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d52217431c3435e8d0adbd1688a61af, 4af4b4a9d34046008efc05e88bd8eedb, 5d1f555bd3b14e7d9cb63be8aa60fcd9; no voters:
I20260811 12:38:19.666154 2547 raft_consensus.cc:2781] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:38:19.666301 2547 raft_consensus.cc:482] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:38:19.666392 2547 raft_consensus.cc:3037] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:19.668725 2547 raft_consensus.cc:504] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.669379 2547 leader_election.cc:288] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [CANDIDATE]: Term 1 election: Requested vote from peers 0d52217431c3435e8d0adbd1688a61af (127.3.204.65:33017), 5d1f555bd3b14e7d9cb63be8aa60fcd9 (127.3.204.67:44407)
I20260811 12:38:19.670003 1900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843201827fc24cd8a28c5961d43548ef" candidate_uuid: "4af4b4a9d34046008efc05e88bd8eedb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d52217431c3435e8d0adbd1688a61af"
I20260811 12:38:19.670003 2402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843201827fc24cd8a28c5961d43548ef" candidate_uuid: "4af4b4a9d34046008efc05e88bd8eedb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9"
I20260811 12:38:19.670207 1900 raft_consensus.cc:3037] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:19.670220 2402 raft_consensus.cc:3037] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:19.672572 1900 raft_consensus.cc:2454] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af4b4a9d34046008efc05e88bd8eedb in term 1.
I20260811 12:38:19.672741 2402 raft_consensus.cc:2454] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af4b4a9d34046008efc05e88bd8eedb in term 1.
I20260811 12:38:19.673085 2033 leader_election.cc:302] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d52217431c3435e8d0adbd1688a61af, 4af4b4a9d34046008efc05e88bd8eedb; no voters:
I20260811 12:38:19.673454 2547 raft_consensus.cc:2781] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:19.674007 2547 raft_consensus.cc:686] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 1 LEADER]: Becoming Leader. State: Replica: 4af4b4a9d34046008efc05e88bd8eedb, State: Running, Role: LEADER
I20260811 12:38:19.674616 2547 consensus_queue.cc:237] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } }
I20260811 12:38:19.680011 1612 catalog_manager.cc:5495] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af4b4a9d34046008efc05e88bd8eedb (127.3.204.66). New cstate: current_term: 1 leader_uuid: "4af4b4a9d34046008efc05e88bd8eedb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af4b4a9d34046008efc05e88bd8eedb" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 } health_report { overall_health: UNKNOWN } } }
I20260811 12:38:19.749737 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.743272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.3.204.66:36483)
0811 12:38:19.743508 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.743514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.743535 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:38:19.743758 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.743763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.743862 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.744010 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:38:19.744474 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.745215 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.746920 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.747386 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.747504 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.748151 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:38:19.748946 (+ 795us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:38:19.749101 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.749276 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.749426 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:38:19.758518 1900 raft_consensus.cc:1261] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 1 FOLLOWER]: Refusing update from remote peer 4af4b4a9d34046008efc05e88bd8eedb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:19.758680 2402 raft_consensus.cc:1261] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 1 FOLLOWER]: Refusing update from remote peer 4af4b4a9d34046008efc05e88bd8eedb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:19.759640 2547 consensus_queue.cc:1035] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d52217431c3435e8d0adbd1688a61af" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 33017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:19.759986 2556 consensus_queue.cc:1035] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1f555bd3b14e7d9cb63be8aa60fcd9" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:19.770928 2564 mvcc.cc:205] Tried to move back new op lower bound from 7317306981399687168 to 7317306981071454208. Current Snapshot: MvccSnapshot[applied={T|T < 7317306981399687168}]
I20260811 12:38:19.785902 1612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:19.786211 1612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:19.789935 1612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaaf516a534e476c8e00812edaa168c0: Sending DeleteTablet for 3 replicas of tablet 843201827fc24cd8a28c5961d43548ef
I20260811 12:38:19.791168 2105 tablet_service.cc:1514] Processing DeleteTablet for tablet 843201827fc24cd8a28c5961d43548ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:19 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:33900
I20260811 12:38:19.791165 2360 tablet_service.cc:1514] Processing DeleteTablet for tablet 843201827fc24cd8a28c5961d43548ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:19 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:60110
I20260811 12:38:19.791404 1860 tablet_service.cc:1514] Processing DeleteTablet for tablet 843201827fc24cd8a28c5961d43548ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:19 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:57778
I20260811 12:38:19.793700 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 1732
I20260811 12:38:19.798072 2589 tablet_replica.cc:330] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: stopping tablet replica
I20260811 12:38:19.800560 2589 raft_consensus.cc:2229] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:19.801348 2585 tablet_replica.cc:330] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: stopping tablet replica
I20260811 12:38:19.801828 2589 raft_consensus.cc:2258] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:19.801848 2585 raft_consensus.cc:2229] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:19.802282 2582 tablet_replica.cc:330] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: stopping tablet replica
I20260811 12:38:19.802423 2585 raft_consensus.cc:2258] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:19.802742 2582 raft_consensus.cc:2229] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:38:19.803442 2582 raft_consensus.cc:2258] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:19.805068 2585 ts_tablet_manager.cc:1905] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:19.805920 2589 ts_tablet_manager.cc:1905] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:19.807899 2582 ts_tablet_manager.cc:1905] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 12:38:19 dist-test-slave-lgr5 krb5kdc[1529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:38:19.812176 2585 ts_tablet_manager.cc:1918] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:19.812469 2585 log.cc:1196] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/843201827fc24cd8a28c5961d43548ef
I20260811 12:38:19.813097 2585 ts_tablet_manager.cc:1939] T 843201827fc24cd8a28c5961d43548ef P 5d1f555bd3b14e7d9cb63be8aa60fcd9: Deleting consensus metadata
I20260811 12:38:19.815207 2589 ts_tablet_manager.cc:1918] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:19.815624 1585 catalog_manager.cc:4841] TS 5d1f555bd3b14e7d9cb63be8aa60fcd9 (127.3.204.67:44407): tablet 843201827fc24cd8a28c5961d43548ef (table test-table [id=ab6234f3890f483586223829cab3367f]) successfully deleted
I20260811 12:38:19.816231 2589 log.cc:1196] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/843201827fc24cd8a28c5961d43548ef
I20260811 12:38:19.816862 2589 ts_tablet_manager.cc:1939] T 843201827fc24cd8a28c5961d43548ef P 0d52217431c3435e8d0adbd1688a61af: Deleting consensus metadata
I20260811 12:38:19.818888 1588 catalog_manager.cc:4841] TS 0d52217431c3435e8d0adbd1688a61af (127.3.204.65:33017): tablet 843201827fc24cd8a28c5961d43548ef (table test-table [id=ab6234f3890f483586223829cab3367f]) successfully deleted
I20260811 12:38:19.819550 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.795256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.3.204.65:33017)
0811 12:38:19.795471 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.795478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.800117 (+ 4639us) server_negotiation.cc:407] Connection header received
0811 12:38:19.800331 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.800337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.800415 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.800521 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:19.802259 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.803357 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.805330 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.805978 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.806078 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.811550 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:19.811565 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:19.811588 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:19.811633 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:19.815754 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:19.816832 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:19.816842 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:19.816850 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:19.817036 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:19.817558 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:19.817564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:19.817567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:19.818072 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:19.818384 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.818998 (+ 614us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.819233 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:38:19.820956 1979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:19.824072 2582 ts_tablet_manager.cc:1918] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:19.824434 2582 log.cc:1196] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/843201827fc24cd8a28c5961d43548ef
I20260811 12:38:19.825196 2582 ts_tablet_manager.cc:1939] T 843201827fc24cd8a28c5961d43548ef P 4af4b4a9d34046008efc05e88bd8eedb: Deleting consensus metadata
I20260811 12:38:19.827337 1587 catalog_manager.cc:4841] TS 4af4b4a9d34046008efc05e88bd8eedb (127.3.204.66:36483): tablet 843201827fc24cd8a28c5961d43548ef (table test-table [id=ab6234f3890f483586223829cab3367f]) successfully deleted
I20260811 12:38:19.910738 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 1732
I20260811 12:38:19.940763 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 1998
Aug 11 12:38:19 dist-test-slave-lgr5 krb5kdc[1529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:38:19.957641 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:19.941509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.3.204.66:36483)
0811 12:38:19.941724 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:19.941731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:19.941755 (+ 24us) server_negotiation.cc:407] Connection header received
0811 12:38:19.941919 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:19.941925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:19.942007 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:19.942093 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:19.943358 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.944268 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.945532 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:19.946041 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:19.946169 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:19.951439 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:19.951457 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:19.951478 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:19.951530 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:19.954620 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:19.955526 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:19.955534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:19.955536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:19.955688 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:19.956071 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:19.956075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:19.956105 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:19.956516 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:19.956789 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:19.957133 (+ 344us) server_negotiation.cc:299] Negotiation successful
0811 12:38:19.957331 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 12:38:19.958398 2228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:20.073383 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 1998
I20260811 12:38:20.100765 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 2247
Aug 11 12:38:20 dist-test-slave-lgr5 krb5kdc[1529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:38:20.117178 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:20.101522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.3.204.67:44407)
0811 12:38:20.101696 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:20.101701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:20.101747 (+ 46us) server_negotiation.cc:407] Connection header received
0811 12:38:20.101955 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:20.101975 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:20.102059 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:20.102160 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:20.103597 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:20.104422 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:20.105941 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:20.106350 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:20.106462 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:20.111328 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:20.111347 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:20.111366 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:20.111413 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:20.114410 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:20.115235 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:20.115243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:20.115246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:20.115401 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:20.115720 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:20.115724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:20.115726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:20.116144 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:20.116380 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:20.116656 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 12:38:20.116827 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:38:20.117810 2494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:20.207545 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 2247
I20260811 12:38:20.228724 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 1549
I20260811 12:38:20.229971 1671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:20.340184 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 1549
2026-08-11T12:38:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4956 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-11T12:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:20Z Disabled control of system clock
I20260811 12:38:20.438300 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37409
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 12:38:20.781582 2599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:20.781889 2599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:20.781980 2599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:20.789314 2599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:20.789443 2599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:20.789489 2599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:20.789546 2599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:20.789589 2599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:20.798959 2599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:37409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:20.800292 2599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:20.802075 2599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:20.813565 2610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:20.813604 2613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:20.813530 2617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:20.814601 2599 server_base.cc:1034] running on GCE node
I20260811 12:38:20.815388 2599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:20.816852 2599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:20.818074 2599 hybrid_clock.cc:648] HybridClock initialized: now 1786451900818023 us; error 43 us; skew 500 ppm
I20260811 12:38:20.818419 2599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:20.820704 2599 webserver.cc:460] Webserver started at http://127.3.204.126:38555/ using document root <none> and password file <none>
I20260811 12:38:20.821408 2599 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:20.821523 2599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:20.821847 2599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:20.824950 2599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "596e6b2e98fb41fc89dec4dbf5c50600"
format_stamp: "Formatted at 2026-08-11 12:38:20 on dist-test-slave-lgr5"
server_key: "765fea1c069f6d01b6ed2268a5bb1a55"
server_key_iv: "b3d7bb03eb336aa442775a9a6a11eb91"
server_key_version: "encryptionkey@0"
I20260811 12:38:20.825877 2599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "596e6b2e98fb41fc89dec4dbf5c50600"
format_stamp: "Formatted at 2026-08-11 12:38:20 on dist-test-slave-lgr5"
server_key: "765fea1c069f6d01b6ed2268a5bb1a55"
server_key_iv: "b3d7bb03eb336aa442775a9a6a11eb91"
server_key_version: "encryptionkey@0"
I20260811 12:38:20.831434 2599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 12:38:20.835130 2628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:20.836520 2599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:38:20.836699 2599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "596e6b2e98fb41fc89dec4dbf5c50600"
format_stamp: "Formatted at 2026-08-11 12:38:20 on dist-test-slave-lgr5"
server_key: "765fea1c069f6d01b6ed2268a5bb1a55"
server_key_iv: "b3d7bb03eb336aa442775a9a6a11eb91"
server_key_version: "encryptionkey@0"
I20260811 12:38:20.836913 2599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:20.848991 2599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:20.849862 2599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:20.850148 2599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:20.868570 2599 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:37409
I20260811 12:38:20.868606 2732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:37409 every 8 connection(s)
I20260811 12:38:20.869922 2599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:20.873662 2733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:20.877254 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 2599
I20260811 12:38:20.877434 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:20.878015 3889 external_mini_cluster.cc:1428] Setting key 5c75c0362cb5472b9cc708428f91307f
I20260811 12:38:20.886315 2733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600: Bootstrap starting.
I20260811 12:38:20.889617 2733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:20.890722 2733 log.cc:824] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:20.891857 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:20.880043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.3.204.126:37409)
0811 12:38:20.881091 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:20.881102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:20.881147 (+ 45us) server_negotiation.cc:407] Connection header received
0811 12:38:20.882847 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:20.882877 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:20.883383 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:20.883806 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:20.884443 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:20.885732 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:20.887150 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:20.887861 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:20.888037 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:20.889266 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:20.889274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:20.889287 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:20.889871 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:20.889965 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:20.889995 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:20.890109 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:20.890320 (+ 211us) server_negotiation.cc:299] Negotiation successful
0811 12:38:20.890636 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":350,"threads_started":1}
I20260811 12:38:20.894006 2733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600: No bootstrap required, opened a new log
I20260811 12:38:20.901546 2733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596e6b2e98fb41fc89dec4dbf5c50600" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37409 } }
I20260811 12:38:20.901939 2733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:20.902045 2733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596e6b2e98fb41fc89dec4dbf5c50600, State: Initialized, Role: FOLLOWER
I20260811 12:38:20.902630 2733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596e6b2e98fb41fc89dec4dbf5c50600" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37409 } }
I20260811 12:38:20.902864 2733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:20.903000 2733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:20.903155 2733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:20.905664 2733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596e6b2e98fb41fc89dec4dbf5c50600" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37409 } }
I20260811 12:38:20.906307 2733 leader_election.cc:302] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 596e6b2e98fb41fc89dec4dbf5c50600; no voters:
I20260811 12:38:20.906965 2733 leader_election.cc:288] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:20.907089 2744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:20.908140 2744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [term 1 LEADER]: Becoming Leader. State: Replica: 596e6b2e98fb41fc89dec4dbf5c50600, State: Running, Role: LEADER
I20260811 12:38:20.908643 2744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596e6b2e98fb41fc89dec4dbf5c50600" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37409 } }
I20260811 12:38:20.909060 2733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:20.911273 2745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "596e6b2e98fb41fc89dec4dbf5c50600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596e6b2e98fb41fc89dec4dbf5c50600" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37409 } } }
I20260811 12:38:20.911655 2745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:20.912480 2746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596e6b2e98fb41fc89dec4dbf5c50600. Latest consensus state: current_term: 1 leader_uuid: "596e6b2e98fb41fc89dec4dbf5c50600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596e6b2e98fb41fc89dec4dbf5c50600" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37409 } } }
I20260811 12:38:20.912789 2746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:20.914243 2754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:20.921420 2754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:20.929687 2754 catalog_manager.cc:1347] Generated new cluster ID: eee765499cfa4fb1a5aa6745ba825ad1
I20260811 12:38:20.929814 2754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:20.951020 2754 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:20.952489 2754 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:20.963132 2754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600: Generated new TSK 0
I20260811 12:38:20.963986 2754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:20.973523 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37409
--builtin_ntp_servers=127.3.204.84:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 12:38:21.322127 2777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:21.322436 2777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:21.322564 2777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:21.330298 2777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:21.330422 2777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:21.330592 2777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:38:21.340242 2777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:21.341674 2777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:21.343580 2777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:21.357106 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:21.357106 2793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:21.357069 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:21.358026 2777 server_base.cc:1034] running on GCE node
I20260811 12:38:21.358654 2777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:21.359699 2777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:21.360900 2777 hybrid_clock.cc:648] HybridClock initialized: now 1786451901360847 us; error 44 us; skew 500 ppm
I20260811 12:38:21.361229 2777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:21.363538 2777 webserver.cc:460] Webserver started at http://127.3.204.65:37109/ using document root <none> and password file <none>
I20260811 12:38:21.364295 2777 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:21.364396 2777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:21.364763 2777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:21.367718 2777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "13ba3d7195f04d81af38609fe82b8386"
format_stamp: "Formatted at 2026-08-11 12:38:21 on dist-test-slave-lgr5"
server_key: "80029154d14a7941300c5bd674636d14"
server_key_iv: "18900d399b3f6540712b8193f73c74f7"
server_key_version: "encryptionkey@0"
I20260811 12:38:21.368530 2777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "13ba3d7195f04d81af38609fe82b8386"
format_stamp: "Formatted at 2026-08-11 12:38:21 on dist-test-slave-lgr5"
server_key: "80029154d14a7941300c5bd674636d14"
server_key_iv: "18900d399b3f6540712b8193f73c74f7"
server_key_version: "encryptionkey@0"
I20260811 12:38:21.373557 2777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 12:38:21.377170 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:21.378486 2777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:38:21.378638 2777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "13ba3d7195f04d81af38609fe82b8386"
format_stamp: "Formatted at 2026-08-11 12:38:21 on dist-test-slave-lgr5"
server_key: "80029154d14a7941300c5bd674636d14"
server_key_iv: "18900d399b3f6540712b8193f73c74f7"
server_key_version: "encryptionkey@0"
I20260811 12:38:21.378852 2777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:21.401038 2777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:21.401938 2777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:21.402266 2777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:21.403311 2777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:21.405017 2777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:21.405120 2777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:21.405236 2777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:21.405308 2777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:21.437532 2777 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:45901
I20260811 12:38:21.437602 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:45901 every 8 connection(s)
I20260811 12:38:21.438915 2777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:38:21.447037 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 2777
I20260811 12:38:21.447224 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:38:21.447737 3889 external_mini_cluster.cc:1428] Setting key aa28bb7efb60536b1a2671fc5e49473e
I20260811 12:38:21.450623 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37409
--builtin_ntp_servers=127.3.204.84:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 12:38:21.452265 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:21.441642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:56927 (local address 127.3.204.126:37409)
0811 12:38:21.442089 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:21.442094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:21.443205 (+ 1111us) server_negotiation.cc:407] Connection header received
0811 12:38:21.445135 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:21.445141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:21.445228 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:21.445327 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:21.446438 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:21.447439 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:21.449128 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:21.449690 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:21.449792 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:21.451453 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:21.451458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:21.451461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:21.451480 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:21.451525 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:21.451538 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:21.451606 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:21.451873 (+ 267us) server_negotiation.cc:299] Negotiation successful
0811 12:38:21.451959 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260811 12:38:21.453699 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:21.442091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37409 (local address 127.3.204.65:56927)
0811 12:38:21.443050 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:21.443095 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:21.444855 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:21.445578 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:21.445586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:21.445675 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:21.446254 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:21.446271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:21.447598 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:21.447602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:21.448985 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:21.449000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:21.449927 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:21.449935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:21.450259 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:21.451048 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:21.451051 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:21.451105 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:21.451221 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:21.451630 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:21.451642 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:21.451651 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 12:38:21.451871 (+ 220us) client_negotiation.cc:240] Negotiation successful
0811 12:38:21.452337 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":263,"threads_started":1}
I20260811 12:38:21.455293 2999 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37409
I20260811 12:38:21.455617 2999 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:21.456444 2999 heartbeater.cc:511] Master 127.3.204.126:37409 requested a full tablet report, sending...
I20260811 12:38:21.458623 2666 ts_manager.cc:194] Registered new tserver with Master: 13ba3d7195f04d81af38609fe82b8386 (127.3.204.65:45901)
I20260811 12:38:21.460191 2666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.65:56927
W20260811 12:38:21.808481 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:21.808807 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:21.808940 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:21.816404 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:21.816530 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:21.816684 3008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:38:21.826004 3008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:21.827389 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:21.829252 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:21.842336 3015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:21.842356 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:21.842411 3017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:21.843341 3008 server_base.cc:1034] running on GCE node
I20260811 12:38:21.843956 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:21.844995 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:21.846264 3008 hybrid_clock.cc:648] HybridClock initialized: now 1786451901846175 us; error 47 us; skew 500 ppm
I20260811 12:38:21.846642 3008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:21.849202 3008 webserver.cc:460] Webserver started at http://127.3.204.66:37067/ using document root <none> and password file <none>
I20260811 12:38:21.849923 3008 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:21.850030 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:21.850364 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:21.853412 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278"
format_stamp: "Formatted at 2026-08-11 12:38:21 on dist-test-slave-lgr5"
server_key: "8a6121909fc8d948273cae4caba9fe5f"
server_key_iv: "9ac63dde293580472d961c2b1afc17a1"
server_key_version: "encryptionkey@0"
I20260811 12:38:21.854236 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278"
format_stamp: "Formatted at 2026-08-11 12:38:21 on dist-test-slave-lgr5"
server_key: "8a6121909fc8d948273cae4caba9fe5f"
server_key_iv: "9ac63dde293580472d961c2b1afc17a1"
server_key_version: "encryptionkey@0"
I20260811 12:38:21.859423 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260811 12:38:21.863104 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:21.864543 3008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:38:21.864708 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278"
format_stamp: "Formatted at 2026-08-11 12:38:21 on dist-test-slave-lgr5"
server_key: "8a6121909fc8d948273cae4caba9fe5f"
server_key_iv: "9ac63dde293580472d961c2b1afc17a1"
server_key_version: "encryptionkey@0"
I20260811 12:38:21.864916 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:21.879802 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:21.880681 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:21.880959 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:21.881959 3008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:21.883406 3008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:21.883524 3008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:21.883630 3008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:21.883698 3008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:21.914127 3008 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:39531
I20260811 12:38:21.914180 3256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:39531 every 8 connection(s)
I20260811 12:38:21.915517 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:38:21.923743 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 3008
I20260811 12:38:21.923933 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:38:21.924571 3889 external_mini_cluster.cc:1428] Setting key a04b0bbab5e2f3620d1684668183d475
I20260811 12:38:21.927783 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37409
--builtin_ntp_servers=127.3.204.84:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 12:38:21.928429 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:21.918034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:47937 (local address 127.3.204.126:37409)
0811 12:38:21.918206 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:21.918211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:21.919541 (+ 1330us) server_negotiation.cc:407] Connection header received
0811 12:38:21.921287 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:21.921291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:21.921357 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:21.921443 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:21.922466 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:21.923188 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:21.925416 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:21.925989 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:21.926102 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:21.927505 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:21.927510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:21.927513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:21.927531 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:21.927566 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:21.927578 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:21.927647 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:21.927908 (+ 261us) server_negotiation.cc:299] Negotiation successful
0811 12:38:21.928144 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":58}
I20260811 12:38:21.929766 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:21.918507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37409 (local address 127.3.204.66:47937)
0811 12:38:21.919383 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:21.919418 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:21.921062 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:21.921657 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:21.921665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:21.921749 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:21.922315 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:21.922338 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:21.923338 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:21.923343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:21.925262 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:21.925276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:21.926197 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:21.926202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:21.926423 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:21.927151 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:21.927155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:21.927210 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:21.927312 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:21.927674 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:21.927685 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:21.927694 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 12:38:21.927899 (+ 205us) client_negotiation.cc:240] Negotiation successful
0811 12:38:21.928369 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":253,"threads_started":1}
I20260811 12:38:21.931260 3257 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37409
I20260811 12:38:21.931567 3257 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:21.932402 3257 heartbeater.cc:511] Master 127.3.204.126:37409 requested a full tablet report, sending...
I20260811 12:38:21.933775 2666 ts_manager.cc:194] Registered new tserver with Master: f6bdc0d8f5c9427fbbc2ad614fb99278 (127.3.204.66:39531)
I20260811 12:38:21.934593 2666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.66:47937
W20260811 12:38:22.272791 3265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:22.273105 3265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:22.273237 3265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:22.281080 3265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:22.281215 3265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:22.281378 3265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:38:22.291323 3265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:22.292778 3265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:22.294567 3265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:22.308213 3283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:22.308145 3284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:22.308106 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:22.309108 3265 server_base.cc:1034] running on GCE node
I20260811 12:38:22.310012 3265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:22.311131 3265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:22.312405 3265 hybrid_clock.cc:648] HybridClock initialized: now 1786451902312321 us; error 49 us; skew 500 ppm
I20260811 12:38:22.312765 3265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:22.315310 3265 webserver.cc:460] Webserver started at http://127.3.204.67:45291/ using document root <none> and password file <none>
I20260811 12:38:22.316074 3265 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:22.316217 3265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:22.316560 3265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:22.319635 3265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "50354a9319854608b0e5b560f13deca7"
format_stamp: "Formatted at 2026-08-11 12:38:22 on dist-test-slave-lgr5"
server_key: "31d53d66b4cf1d88ac77865618dddb29"
server_key_iv: "d16d813b125616024f576e7fc19fa42f"
server_key_version: "encryptionkey@0"
I20260811 12:38:22.320531 3265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "50354a9319854608b0e5b560f13deca7"
format_stamp: "Formatted at 2026-08-11 12:38:22 on dist-test-slave-lgr5"
server_key: "31d53d66b4cf1d88ac77865618dddb29"
server_key_iv: "d16d813b125616024f576e7fc19fa42f"
server_key_version: "encryptionkey@0"
I20260811 12:38:22.326061 3265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 12:38:22.330281 3294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:22.331763 3265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:38:22.331949 3265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "50354a9319854608b0e5b560f13deca7"
format_stamp: "Formatted at 2026-08-11 12:38:22 on dist-test-slave-lgr5"
server_key: "31d53d66b4cf1d88ac77865618dddb29"
server_key_iv: "d16d813b125616024f576e7fc19fa42f"
server_key_version: "encryptionkey@0"
I20260811 12:38:22.332242 3265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:22.347615 3265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:22.348598 3265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:22.348912 3265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:22.349973 3265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:22.351529 3265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:22.351640 3265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:22.351749 3265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:22.351820 3265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:22.384963 3265 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:33263
I20260811 12:38:22.385022 3509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:33263 every 8 connection(s)
I20260811 12:38:22.386307 3265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:38:22.389284 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 3265
I20260811 12:38:22.389465 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:38:22.390051 3889 external_mini_cluster.cc:1428] Setting key 1bff174c9ee537a2865dac7c32f7f103
I20260811 12:38:22.402860 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.389013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:34791 (local address 127.3.204.126:37409)
0811 12:38:22.389178 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.389182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.390988 (+ 1806us) server_negotiation.cc:407] Connection header received
0811 12:38:22.393345 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.393351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.393433 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.393518 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:22.394592 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.395363 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.397023 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.397463 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.397573 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.402032 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:22.402037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:22.402040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:22.402060 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:22.402093 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:22.402106 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:22.402181 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.402411 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.402520 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 12:38:22.404060 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.389749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:37409 (local address 127.3.204.67:34791)
0811 12:38:22.390833 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.390870 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:22.393093 (+ 2223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:22.393763 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:22.393771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:22.393860 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:22.394421 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.394438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.395518 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.395522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.396878 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.396891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.397695 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.397704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.397928 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.401689 (+ 3761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:22.401692 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:22.401735 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:22.401821 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:22.402195 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:22.402206 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:22.402214 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 12:38:22.402369 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 12:38:22.402811 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":328,"threads_started":1}
I20260811 12:38:22.405488 3510 heartbeater.cc:347] Connected to a master server at 127.3.204.126:37409
I20260811 12:38:22.405778 3510 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:22.406555 3510 heartbeater.cc:511] Master 127.3.204.126:37409 requested a full tablet report, sending...
I20260811 12:38:22.407909 2664 ts_manager.cc:194] Registered new tserver with Master: 50354a9319854608b0e5b560f13deca7 (127.3.204.67:33263)
I20260811 12:38:22.408807 2664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.67:34791
I20260811 12:38:22.417759 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:38:22.427606 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.421808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.3.204.126:37409)
0811 12:38:22.422024 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.422028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.422393 (+ 365us) server_negotiation.cc:407] Connection header received
0811 12:38:22.422535 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.422539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.422636 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.422720 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:22.423254 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.424003 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.425459 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.425912 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.426011 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.426993 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:22.426998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:22.427001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:22.427021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:22.427057 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:22.427070 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:22.427144 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.427255 (+ 111us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.427338 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:38:22.432067 2664 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:38:22.439522 2664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:38:22.463478 2999 heartbeater.cc:503] Master 127.3.204.126:37409 was elected leader, sending a full tablet report...
I20260811 12:38:22.475842 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.455543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:39531 (local address 127.0.0.1:52004)
0811 12:38:22.467305 (+ 11762us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.467323 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:22.467552 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:22.468406 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:22.468410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:22.468435 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:22.468986 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.468999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.471002 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.471008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.473068 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.473081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.474939 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.474946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.475124 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.475156 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 12:38:22.475249 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 12:38:22.475385 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11629,"mutex_wait_us":114,"thread_start_us":7471,"threads_started":1}
I20260811 12:38:22.475905 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.453055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.3.204.65:45901)
0811 12:38:22.466492 (+ 13437us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.466500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.466531 (+ 31us) server_negotiation.cc:407] Connection header received
0811 12:38:22.466625 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.466633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.466882 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.467099 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:22.468156 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.469581 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.472527 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.474042 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.474184 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.474504 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.475501 (+ 997us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.475650 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13264,"thread_start_us":13130,"threads_started":1}
I20260811 12:38:22.476758 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.453034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.3.204.67:33263)
0811 12:38:22.466191 (+ 13157us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.466202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.466245 (+ 43us) server_negotiation.cc:407] Connection header received
0811 12:38:22.466361 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.466368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.466663 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.468581 (+ 1918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:22.469314 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.471456 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.473878 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.475293 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.475414 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.475672 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.476333 (+ 661us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.476474 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12949,"thread_start_us":12763,"threads_started":1}
I20260811 12:38:22.476760 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.452660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:33263 (local address 127.0.0.1:33450)
0811 12:38:22.465145 (+ 12485us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.465201 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:22.465413 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:22.468545 (+ 3132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:22.468549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:22.468586 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:22.469090 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.469103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.471628 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.471633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.473660 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.473675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.475817 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.475828 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.476005 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.476175 (+ 170us) client_negotiation.cc:769] Sending connection context
0811 12:38:22.476271 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 12:38:22.476409 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12166,"mutex_wait_us":10,"spinlock_wait_cycles":29952,"thread_start_us":14347,"threads_started":1}
I20260811 12:38:22.476924 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.456230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.3.204.66:39531)
0811 12:38:22.464478 (+ 8248us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.464503 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.467432 (+ 2929us) server_negotiation.cc:407] Connection header received
0811 12:38:22.467768 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.467776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.468003 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.468198 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:22.469154 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.470866 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.473256 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.474400 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.474544 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.474784 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.476377 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.476578 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8050,"thread_start_us":7924,"threads_started":1}
I20260811 12:38:22.478230 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.452990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:45901 (local address 127.0.0.1:34990)
0811 12:38:22.465145 (+ 12155us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.465201 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:22.465413 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:22.467386 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:22.467394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:22.467422 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:22.467921 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.467936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.469768 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.469773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.472280 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.472298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.474640 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.474648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.474871 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.474934 (+ 63us) client_negotiation.cc:769] Sending connection context
0811 12:38:22.475091 (+ 157us) client_negotiation.cc:240] Negotiation successful
0811 12:38:22.475304 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11867,"mutex_wait_us":3,"spinlock_wait_cycles":26752,"thread_start_us":11542,"threads_started":1}
I20260811 12:38:22.480985 3385 tablet_service.cc:1467] Processing CreateTablet for tablet ddf56069b63f437c8d28a8e6917b38ad (DEFAULT_TABLE table=test-table [id=c46939cf0d194b0f86f39046e26be6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:22.480908 3139 tablet_service.cc:1467] Processing CreateTablet for tablet ddf56069b63f437c8d28a8e6917b38ad (DEFAULT_TABLE table=test-table [id=c46939cf0d194b0f86f39046e26be6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:22.482594 3385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf56069b63f437c8d28a8e6917b38ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:22.482883 3139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf56069b63f437c8d28a8e6917b38ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:22.482781 2888 tablet_service.cc:1467] Processing CreateTablet for tablet ddf56069b63f437c8d28a8e6917b38ad (DEFAULT_TABLE table=test-table [id=c46939cf0d194b0f86f39046e26be6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:22.484270 2888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf56069b63f437c8d28a8e6917b38ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:22.494894 3545 tablet_bootstrap.cc:492] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: Bootstrap starting.
I20260811 12:38:22.497754 3547 tablet_bootstrap.cc:492] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: Bootstrap starting.
I20260811 12:38:22.499204 3548 tablet_bootstrap.cc:492] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386: Bootstrap starting.
I20260811 12:38:22.499316 3545 tablet_bootstrap.cc:654] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:22.500738 3545 log.cc:824] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:22.502434 3548 tablet_bootstrap.cc:654] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:22.502455 3547 tablet_bootstrap.cc:654] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:22.503386 3548 log.cc:824] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:22.503657 3547 log.cc:824] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:22.504599 3545 tablet_bootstrap.cc:492] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: No bootstrap required, opened a new log
I20260811 12:38:22.504946 3545 ts_tablet_manager.cc:1397] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260811 12:38:22.506623 3547 tablet_bootstrap.cc:492] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: No bootstrap required, opened a new log
I20260811 12:38:22.506850 3548 tablet_bootstrap.cc:492] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386: No bootstrap required, opened a new log
I20260811 12:38:22.506901 3547 ts_tablet_manager.cc:1397] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260811 12:38:22.507131 3548 ts_tablet_manager.cc:1397] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 12:38:22.513254 3545 raft_consensus.cc:348] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.513711 3545 raft_consensus.cc:374] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:22.513840 3545 raft_consensus.cc:729] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50354a9319854608b0e5b560f13deca7, State: Initialized, Role: FOLLOWER
I20260811 12:38:22.514550 3545 consensus_queue.cc:260] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.515748 3510 heartbeater.cc:503] Master 127.3.204.126:37409 was elected leader, sending a full tablet report...
I20260811 12:38:22.516463 3545 ts_tablet_manager.cc:1428] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260811 12:38:22.517620 3547 raft_consensus.cc:348] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.517623 3548 raft_consensus.cc:348] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.518111 3547 raft_consensus.cc:374] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:22.518110 3548 raft_consensus.cc:374] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:22.518214 3548 raft_consensus.cc:729] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13ba3d7195f04d81af38609fe82b8386, State: Initialized, Role: FOLLOWER
I20260811 12:38:22.518325 3547 raft_consensus.cc:729] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bdc0d8f5c9427fbbc2ad614fb99278, State: Initialized, Role: FOLLOWER
I20260811 12:38:22.518980 3548 consensus_queue.cc:260] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.519166 3547 consensus_queue.cc:260] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.521219 3548 ts_tablet_manager.cc:1428] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260811 12:38:22.523125 3257 heartbeater.cc:503] Master 127.3.204.126:37409 was elected leader, sending a full tablet report...
I20260811 12:38:22.523492 3547 ts_tablet_manager.cc:1428] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260811 12:38:22.554224 3554 raft_consensus.cc:482] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:22.554524 3554 raft_consensus.cc:504] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.556313 3554 leader_election.cc:288] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50354a9319854608b0e5b560f13deca7 (127.3.204.67:33263), f6bdc0d8f5c9427fbbc2ad614fb99278 (127.3.204.66:39531)
I20260811 12:38:22.563443 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.556915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:39299 (local address 127.3.204.66:39531)
0811 12:38:22.557129 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.557137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.557803 (+ 666us) server_negotiation.cc:407] Connection header received
0811 12:38:22.558008 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.558014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.558094 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.558164 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:22.558818 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.559527 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.561696 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.562498 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.562657 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.562750 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.563027 (+ 277us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.563173 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 12:38:22.563444 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.556936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:45313 (local address 127.3.204.67:33263)
0811 12:38:22.557132 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.557137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.557905 (+ 768us) server_negotiation.cc:407] Connection header received
0811 12:38:22.558050 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.558054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.558114 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.558199 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:22.558882 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.559599 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.561678 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.562511 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.562679 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.562753 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.563092 (+ 339us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.563223 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 12:38:22.563608 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.556842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:33263 (local address 127.3.204.65:45313)
0811 12:38:22.557767 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.557786 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:22.557921 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:22.558226 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:22.558231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:22.558253 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:22.558703 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.558712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.559732 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.559736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.561459 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.561478 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.562791 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.562799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.562924 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.562950 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 12:38:22.563045 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 12:38:22.563171 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":96,"thread_start_us":457,"threads_started":1}
I20260811 12:38:22.563671 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.556791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:39531 (local address 127.3.204.65:39299)
0811 12:38:22.557679 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.557695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:22.557840 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:22.558208 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:22.558212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:22.558236 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:22.558671 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.558681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.559657 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.559662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.561490 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:22.561504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.562706 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.562713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:22.562867 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.562893 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 12:38:22.562974 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 12:38:22.563110 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":111,"thread_start_us":557,"threads_started":1}
I20260811 12:38:22.564829 3174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf56069b63f437c8d28a8e6917b38ad" candidate_uuid: "13ba3d7195f04d81af38609fe82b8386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" is_pre_election: true
I20260811 12:38:22.564838 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf56069b63f437c8d28a8e6917b38ad" candidate_uuid: "13ba3d7195f04d81af38609fe82b8386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50354a9319854608b0e5b560f13deca7" is_pre_election: true
I20260811 12:38:22.565486 3430 raft_consensus.cc:2454] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13ba3d7195f04d81af38609fe82b8386 in term 0.
I20260811 12:38:22.565480 3174 raft_consensus.cc:2454] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13ba3d7195f04d81af38609fe82b8386 in term 0.
I20260811 12:38:22.566413 2804 leader_election.cc:302] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13ba3d7195f04d81af38609fe82b8386, 50354a9319854608b0e5b560f13deca7, f6bdc0d8f5c9427fbbc2ad614fb99278; no voters:
I20260811 12:38:22.566887 3554 raft_consensus.cc:2781] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:38:22.567045 3554 raft_consensus.cc:482] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:38:22.567145 3554 raft_consensus.cc:3037] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:22.569773 3554 raft_consensus.cc:504] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.570379 3554 leader_election.cc:288] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [CANDIDATE]: Term 1 election: Requested vote from peers 50354a9319854608b0e5b560f13deca7 (127.3.204.67:33263), f6bdc0d8f5c9427fbbc2ad614fb99278 (127.3.204.66:39531)
I20260811 12:38:22.570884 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf56069b63f437c8d28a8e6917b38ad" candidate_uuid: "13ba3d7195f04d81af38609fe82b8386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50354a9319854608b0e5b560f13deca7"
I20260811 12:38:22.570899 3174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf56069b63f437c8d28a8e6917b38ad" candidate_uuid: "13ba3d7195f04d81af38609fe82b8386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278"
I20260811 12:38:22.571125 3174 raft_consensus.cc:3037] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:22.571126 3430 raft_consensus.cc:3037] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:22.574481 3430 raft_consensus.cc:2454] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13ba3d7195f04d81af38609fe82b8386 in term 1.
I20260811 12:38:22.574476 3174 raft_consensus.cc:2454] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13ba3d7195f04d81af38609fe82b8386 in term 1.
I20260811 12:38:22.574957 2808 leader_election.cc:302] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13ba3d7195f04d81af38609fe82b8386, 50354a9319854608b0e5b560f13deca7; no voters:
I20260811 12:38:22.575261 3554 raft_consensus.cc:2781] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:22.575737 3554 raft_consensus.cc:686] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 1 LEADER]: Becoming Leader. State: Replica: 13ba3d7195f04d81af38609fe82b8386, State: Running, Role: LEADER
I20260811 12:38:22.576280 3554 consensus_queue.cc:237] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } }
I20260811 12:38:22.581164 2664 catalog_manager.cc:5495] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13ba3d7195f04d81af38609fe82b8386 (127.3.204.65). New cstate: current_term: 1 leader_uuid: "13ba3d7195f04d81af38609fe82b8386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13ba3d7195f04d81af38609fe82b8386" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 45901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 } health_report { overall_health: UNKNOWN } } }
I20260811 12:38:22.605243 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.599766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.3.204.65:45901)
0811 12:38:22.599930 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.599935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.599954 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:38:22.600141 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.600145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.600265 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.600393 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:38:22.600793 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.601508 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.603000 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.603434 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.603542 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.604072 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:38:22.604702 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:38:22.604822 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.604910 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.604984 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 12:38:22.614025 3430 raft_consensus.cc:1261] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 1 FOLLOWER]: Refusing update from remote peer 13ba3d7195f04d81af38609fe82b8386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:22.614024 3174 raft_consensus.cc:1261] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 1 FOLLOWER]: Refusing update from remote peer 13ba3d7195f04d81af38609fe82b8386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:22.615064 3568 consensus_queue.cc:1035] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bdc0d8f5c9427fbbc2ad614fb99278" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 39531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:22.615437 3554 consensus_queue.cc:1035] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50354a9319854608b0e5b560f13deca7" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:22.628068 3572 mvcc.cc:205] Tried to move back new op lower bound from 7317306993096548352 to 7317306992956071936. Current Snapshot: MvccSnapshot[applied={T|T < 7317306993096548352}]
I20260811 12:38:22.631726 3573 mvcc.cc:205] Tried to move back new op lower bound from 7317306993096548352 to 7317306992956071936. Current Snapshot: MvccSnapshot[applied={T|T < 7317306993096548352}]
I20260811 12:38:22.642791 2664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:22.643023 2664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:22.646876 2664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 596e6b2e98fb41fc89dec4dbf5c50600: Sending DeleteTablet for 3 replicas of tablet ddf56069b63f437c8d28a8e6917b38ad
I20260811 12:38:22.648136 3385 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf56069b63f437c8d28a8e6917b38ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:22 UTC) from {username='slave'} at 127.0.0.1:33450
I20260811 12:38:22.648114 2888 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf56069b63f437c8d28a8e6917b38ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:22 UTC) from {username='slave'} at 127.0.0.1:34990
I20260811 12:38:22.648429 3139 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf56069b63f437c8d28a8e6917b38ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:22 UTC) from {username='slave'} at 127.0.0.1:52004
I20260811 12:38:22.654325 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 2777
I20260811 12:38:22.677910 3586 tablet_replica.cc:330] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: stopping tablet replica
I20260811 12:38:22.683600 3586 raft_consensus.cc:2229] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:22.684254 3586 raft_consensus.cc:2258] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:22.685089 3588 tablet_replica.cc:330] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386: stopping tablet replica
I20260811 12:38:22.685173 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.656230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.3.204.65:45901)
0811 12:38:22.656454 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.656462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.674692 (+ 18230us) server_negotiation.cc:407] Connection header received
0811 12:38:22.675322 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.675329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.675483 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.675580 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:22.676397 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.677843 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.679999 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.680824 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.680938 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.683027 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:22.683041 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:22.683065 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:22.684022 (+ 957us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:22.684124 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:22.684168 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:22.684279 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.684760 (+ 481us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.684881 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 12:38:22.685526 3588 raft_consensus.cc:2229] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:38:22.682813 3589 tablet_replica.cc:330] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: stopping tablet replica
I20260811 12:38:22.686242 3588 raft_consensus.cc:2258] T ddf56069b63f437c8d28a8e6917b38ad P 13ba3d7195f04d81af38609fe82b8386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:22.686353 2993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:22.686465 3589 raft_consensus.cc:2229] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:22.686707 3586 ts_tablet_manager.cc:1905] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:22.686941 3589 raft_consensus.cc:2258] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:22.689393 3589 ts_tablet_manager.cc:1905] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:22.694217 3586 ts_tablet_manager.cc:1918] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:22.694386 3586 log.cc:1196] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/ddf56069b63f437c8d28a8e6917b38ad
I20260811 12:38:22.694831 3586 ts_tablet_manager.cc:1939] T ddf56069b63f437c8d28a8e6917b38ad P 50354a9319854608b0e5b560f13deca7: Deleting consensus metadata
I20260811 12:38:22.696540 2634 catalog_manager.cc:4841] TS 50354a9319854608b0e5b560f13deca7 (127.3.204.67:33263): tablet ddf56069b63f437c8d28a8e6917b38ad (table test-table [id=c46939cf0d194b0f86f39046e26be6a3]) successfully deleted
I20260811 12:38:22.696888 3589 ts_tablet_manager.cc:1918] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:22.697067 3589 log.cc:1196] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/ddf56069b63f437c8d28a8e6917b38ad
I20260811 12:38:22.697614 3589 ts_tablet_manager.cc:1939] T ddf56069b63f437c8d28a8e6917b38ad P f6bdc0d8f5c9427fbbc2ad614fb99278: Deleting consensus metadata
I20260811 12:38:22.699512 2631 catalog_manager.cc:4841] TS f6bdc0d8f5c9427fbbc2ad614fb99278 (127.3.204.66:39531): tablet ddf56069b63f437c8d28a8e6917b38ad (table test-table [id=c46939cf0d194b0f86f39046e26be6a3]) successfully deleted
I20260811 12:38:22.787838 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 2777
W20260811 12:38:22.814174 2629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.65:45901 (error 108)
I20260811 12:38:22.814291 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 3008
I20260811 12:38:22.816154 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.815671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:45901 (local address 127.0.0.1:35002)
0811 12:38:22.815851 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.815974 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260811 12:38:22.816520 2629 catalog_manager.cc:4584] TS 13ba3d7195f04d81af38609fe82b8386 (127.3.204.65:45901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddf56069b63f437c8d28a8e6917b38ad: Network error: Client connection negotiation failed: client connection to 127.3.204.65:45901: connect: Connection refused (error 111)
I20260811 12:38:22.821210 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.815151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.3.204.66:39531)
0811 12:38:22.815348 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.815353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.815376 (+ 23us) server_negotiation.cc:407] Connection header received
0811 12:38:22.815534 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.815538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.815629 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.815717 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:22.816184 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.816983 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.818297 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.818728 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.818832 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.819787 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:22.819795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:22.819810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:22.820508 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:22.820557 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:22.820583 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:22.820673 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.820843 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.820936 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 12:38:22.821952 3245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:22.870585 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.870006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:45901 (local address 127.0.0.1:35008)
0811 12:38:22.870248 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.870387 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 12:38:22.871783 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.871353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:45901 (local address 127.0.0.1:35018)
0811 12:38:22.871510 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.871621 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 12:38:22.911499 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 3008
I20260811 12:38:22.932376 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.931846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:45901 (local address 127.0.0.1:35026)
0811 12:38:22.932071 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.932194 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260811 12:38:22.933398 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.933057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:45901 (local address 127.0.0.1:35042)
0811 12:38:22.933204 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:22.933273 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 12:38:22.934031 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 3265
I20260811 12:38:22.940907 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:22.934737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.3.204.67:33263)
0811 12:38:22.934927 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:22.934932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:22.934953 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:38:22.935082 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:22.935087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:22.935186 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:22.935273 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:22.935733 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.936574 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.937910 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:22.938340 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:22.938472 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:22.939548 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:22.939558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:22.939573 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:22.940205 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:22.940263 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:22.940286 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:22.940387 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:22.940512 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 12:38:22.940624 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:38:22.941536 3496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:23.014146 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:23.013662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:45901 (local address 127.0.0.1:35058)
0811 12:38:23.013870 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:23.013990 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260811 12:38:23.015277 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:23.014888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:45901 (local address 127.0.0.1:35062)
0811 12:38:23.015092 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:23.015167 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260811 12:38:23.031667 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 3265
I20260811 12:38:23.058008 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 2599
I20260811 12:38:23.059185 2723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:23.169744 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 2599
2026-08-11T12:38:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2830 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:23 dist-test-slave-lgr5 krb5kdc[3603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:23 dist-test-slave-lgr5 krb5kdc[3603](info): set up 2 sockets
Aug 11 12:38:23 dist-test-slave-lgr5 krb5kdc[3603](info): commencing operation
krb5kdc: starting...
W20260811 12:38:24.476898 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.005s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:24 dist-test-slave-lgr5 krb5kdc[3603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:38:24.785187 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:43459
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:43459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:25.132777 3636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:25.133092 3636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:25.133229 3636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:25.140991 3636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:25.141116 3636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:25.141184 3636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:25.141266 3636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:25.141326 3636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:25.151831 3636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:43459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:43459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:25.153323 3636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:25.154762 3636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:25.167289 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:25.167263 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:25.167536 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:25.168272 3636 server_base.cc:1034] running on GCE node
I20260811 12:38:25.169217 3636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:25.170794 3636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:25.172025 3636 hybrid_clock.cc:648] HybridClock initialized: now 1786451905171968 us; error 48 us; skew 500 ppm
I20260811 12:38:25.172468 3636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:25 dist-test-slave-lgr5 krb5kdc[3603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451905, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:25.175607 3636 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:38:25.177153 3636 webserver.cc:460] Webserver started at http://127.3.204.126:38333/ using document root <none> and password file <none>
I20260811 12:38:25.177865 3636 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:25.177994 3636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:25.178315 3636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:25.181562 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "36892538e4084b72881f73c4bb83d096"
format_stamp: "Formatted at 2026-08-11 12:38:25 on dist-test-slave-lgr5"
server_key: "054c68db852c5c3f6005727ed0c9a90a"
server_key_iv: "ee18533a035d7a8b156188d062d7fbdd"
server_key_version: "encryptionkey@0"
I20260811 12:38:25.182343 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "36892538e4084b72881f73c4bb83d096"
format_stamp: "Formatted at 2026-08-11 12:38:25 on dist-test-slave-lgr5"
server_key: "054c68db852c5c3f6005727ed0c9a90a"
server_key_iv: "ee18533a035d7a8b156188d062d7fbdd"
server_key_version: "encryptionkey@0"
I20260811 12:38:25.187940 3636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260811 12:38:25.191994 3662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:25.193518 3636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:38:25.193718 3636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "36892538e4084b72881f73c4bb83d096"
format_stamp: "Formatted at 2026-08-11 12:38:25 on dist-test-slave-lgr5"
server_key: "054c68db852c5c3f6005727ed0c9a90a"
server_key_iv: "ee18533a035d7a8b156188d062d7fbdd"
server_key_version: "encryptionkey@0"
I20260811 12:38:25.193948 3636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:25.209738 3636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:25.214354 3636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:25.214668 3636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:25.233435 3636 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:43459
I20260811 12:38:25.233490 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:43459 every 8 connection(s)
I20260811 12:38:25.234922 3636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:25.235605 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 3636
I20260811 12:38:25.235790 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:25.236404 3889 external_mini_cluster.cc:1428] Setting key 2f6642f1af0676154a2f5854fae38320
I20260811 12:38:25.239537 3762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 12:38:25 dist-test-slave-lgr5 krb5kdc[3603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:25.253443 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096: Bootstrap starting.
I20260811 12:38:25.257694 3762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:25.259063 3762 log.cc:824] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:25.262749 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096: No bootstrap required, opened a new log
I20260811 12:38:25.263686 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:25.238348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.3.204.126:43459)
0811 12:38:25.239308 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:25.239326 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:25.239931 (+ 605us) server_negotiation.cc:407] Connection header received
0811 12:38:25.242314 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:25.242343 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:25.242871 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:25.243339 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:25.245124 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:25.247206 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:25.248765 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:25.249869 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:25.250095 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:25.254388 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:25.254399 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:25.254420 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:25.254488 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:25.258596 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:25.259450 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:25.259459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:25.259468 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:25.259719 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:25.260280 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:25.260285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:25.260287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:25.261036 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:25.261362 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:25.261759 (+ 397us) server_negotiation.cc:299] Negotiation successful
0811 12:38:25.262218 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":380,"threads_started":1}
I20260811 12:38:25.271382 3762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36892538e4084b72881f73c4bb83d096" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43459 } }
I20260811 12:38:25.271854 3762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:25.272004 3762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36892538e4084b72881f73c4bb83d096, State: Initialized, Role: FOLLOWER
I20260811 12:38:25.272639 3762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36892538e4084b72881f73c4bb83d096" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43459 } }
I20260811 12:38:25.272871 3762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:25.273003 3762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:25.273161 3762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:25.275368 3762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36892538e4084b72881f73c4bb83d096" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43459 } }
I20260811 12:38:25.275892 3762 leader_election.cc:302] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36892538e4084b72881f73c4bb83d096; no voters:
I20260811 12:38:25.276603 3762 leader_election.cc:288] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:25.276713 3779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:25.277359 3779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [term 1 LEADER]: Becoming Leader. State: Replica: 36892538e4084b72881f73c4bb83d096, State: Running, Role: LEADER
I20260811 12:38:25.277982 3779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36892538e4084b72881f73c4bb83d096" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43459 } }
I20260811 12:38:25.279028 3762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:25.281189 3780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36892538e4084b72881f73c4bb83d096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36892538e4084b72881f73c4bb83d096" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43459 } } }
I20260811 12:38:25.281610 3780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:25.284644 3783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36892538e4084b72881f73c4bb83d096. Latest consensus state: current_term: 1 leader_uuid: "36892538e4084b72881f73c4bb83d096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36892538e4084b72881f73c4bb83d096" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 43459 } } }
I20260811 12:38:25.285029 3783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:25.285123 3798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:25.289640 3798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:25.298940 3798 catalog_manager.cc:1347] Generated new cluster ID: a0c6bc4a040e4aa597bdd0be3fc169d5
I20260811 12:38:25.299104 3798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:25.327229 3798 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:25.328712 3798 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:25.338500 3798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096: Generated new TSK 0
I20260811 12:38:25.339471 3798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:38:25.425535 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43459
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:25.772574 3818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:25.772876 3818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:25.772975 3818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:25.780763 3818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:25.780900 3818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:25.781014 3818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:38:25.791338 3818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:25.792727 3818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:25.794028 3818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:25.807792 3830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:25.807758 3835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:25.807834 3832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:25.808593 3818 server_base.cc:1034] running on GCE node
I20260811 12:38:25.809244 3818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:25.810230 3818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:25.811431 3818 hybrid_clock.cc:648] HybridClock initialized: now 1786451905811380 us; error 46 us; skew 500 ppm
I20260811 12:38:25.811748 3818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:25 dist-test-slave-lgr5 krb5kdc[3603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451905, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:25.814908 3818 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:38:25.816387 3818 webserver.cc:460] Webserver started at http://127.3.204.65:44457/ using document root <none> and password file <none>
I20260811 12:38:25.817083 3818 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:25.817207 3818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:25.817546 3818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:25.820631 3818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "92225c6dc01d4a3090358db670a12d87"
format_stamp: "Formatted at 2026-08-11 12:38:25 on dist-test-slave-lgr5"
server_key: "270c4b5bc51ff04ee41f2358e5f8f35e"
server_key_iv: "2f7e092212da80c1d0e09591d0be9788"
server_key_version: "encryptionkey@0"
I20260811 12:38:25.821430 3818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "92225c6dc01d4a3090358db670a12d87"
format_stamp: "Formatted at 2026-08-11 12:38:25 on dist-test-slave-lgr5"
server_key: "270c4b5bc51ff04ee41f2358e5f8f35e"
server_key_iv: "2f7e092212da80c1d0e09591d0be9788"
server_key_version: "encryptionkey@0"
I20260811 12:38:25.827103 3818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 12:38:25.830794 3852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:25.832217 3818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:38:25.832384 3818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "92225c6dc01d4a3090358db670a12d87"
format_stamp: "Formatted at 2026-08-11 12:38:25 on dist-test-slave-lgr5"
server_key: "270c4b5bc51ff04ee41f2358e5f8f35e"
server_key_iv: "2f7e092212da80c1d0e09591d0be9788"
server_key_version: "encryptionkey@0"
I20260811 12:38:25.832675 3818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:25.843947 3818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:25.848405 3818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:25.848690 3818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:25.849673 3818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:25.851042 3818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:25.851119 3818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:25.851238 3818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:25.851292 3818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:25.882500 3818 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:41985
I20260811 12:38:25.882522 4080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:41985 every 8 connection(s)
I20260811 12:38:25.883955 3818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:38:25.886160 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 3818
I20260811 12:38:25.886314 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:38:25.886806 3889 external_mini_cluster.cc:1428] Setting key 0d266171ef35da64ce350972cfd2d974
Aug 11 12:38:25 dist-test-slave-lgr5 krb5kdc[3603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451905, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:25.905561 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:25.886609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:50673 (local address 127.3.204.126:43459)
0811 12:38:25.887121 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:25.887127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:25.888072 (+ 945us) server_negotiation.cc:407] Connection header received
0811 12:38:25.890439 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:25.890443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:25.890513 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:25.890600 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:25.892533 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:25.893229 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:25.895269 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:25.895687 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:25.895789 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:25.899966 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:25.899972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:25.899975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:25.900008 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:25.902473 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:25.903215 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:25.903219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:25.903222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:25.903344 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:25.903777 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:25.903780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:25.903783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:25.904053 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:25.904222 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:25.904962 (+ 740us) server_negotiation.cc:299] Negotiation successful
0811 12:38:25.905199 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":237,"threads_started":1}
I20260811 12:38:25.906719 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:25.887121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43459 (local address 127.3.204.65:50673)
0811 12:38:25.887934 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:25.887973 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:25.890127 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:25.890855 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:25.890866 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:25.891553 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:25.892344 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:25.892369 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:25.893374 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:25.893379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:25.895087 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:25.895113 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:25.895916 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:25.895923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:25.896235 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:25.896968 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:25.896997 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:25.899727 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:25.902629 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:25.902639 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:25.902657 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:25.903087 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:25.903449 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:25.903454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:25.903458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:25.903652 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:25.904205 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:25.904213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:25.904573 (+ 360us) client_negotiation.cc:769] Sending connection context
0811 12:38:25.904822 (+ 249us) client_negotiation.cc:240] Negotiation successful
0811 12:38:25.905426 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":321,"threads_started":1}
I20260811 12:38:25.908622 4081 heartbeater.cc:347] Connected to a master server at 127.3.204.126:43459
I20260811 12:38:25.909000 4081 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:25.909886 4081 heartbeater.cc:511] Master 127.3.204.126:43459 requested a full tablet report, sending...
I20260811 12:38:25.912405 3701 ts_manager.cc:194] Registered new tserver with Master: 92225c6dc01d4a3090358db670a12d87 (127.3.204.65:41985)
I20260811 12:38:25.914300 3701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:50673
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:38:25.956295 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43459
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:26.301638 4090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:26.301960 4090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:26.302058 4090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:26.309413 4090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:26.309543 4090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:26.309656 4090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:38:26.320036 4090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:26.321444 4090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:26.322844 4090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:26.336778 4096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:26.337029 4097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:26.337050 4105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:26.338059 4090 server_base.cc:1034] running on GCE node
I20260811 12:38:26.338708 4090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:26.339783 4090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:26.341002 4090 hybrid_clock.cc:648] HybridClock initialized: now 1786451906340935 us; error 59 us; skew 500 ppm
I20260811 12:38:26.341338 4090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:26 dist-test-slave-lgr5 krb5kdc[3603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451906, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:26.344660 4090 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:38:26.346321 4090 webserver.cc:460] Webserver started at http://127.3.204.66:40027/ using document root <none> and password file <none>
I20260811 12:38:26.347044 4090 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:26.347159 4090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:26.347496 4090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:26.350660 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "1e6c9b35ae5944919c1ba29aaa862880"
format_stamp: "Formatted at 2026-08-11 12:38:26 on dist-test-slave-lgr5"
server_key: "9d1fab92ba1ea7ab7154edcea103605a"
server_key_iv: "0aa96f0f48e2b3af532290b6b9cb69c3"
server_key_version: "encryptionkey@0"
I20260811 12:38:26.351478 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "1e6c9b35ae5944919c1ba29aaa862880"
format_stamp: "Formatted at 2026-08-11 12:38:26 on dist-test-slave-lgr5"
server_key: "9d1fab92ba1ea7ab7154edcea103605a"
server_key_iv: "0aa96f0f48e2b3af532290b6b9cb69c3"
server_key_version: "encryptionkey@0"
I20260811 12:38:26.357173 4090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260811 12:38:26.361020 4114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:26.362418 4090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:26.362588 4090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "1e6c9b35ae5944919c1ba29aaa862880"
format_stamp: "Formatted at 2026-08-11 12:38:26 on dist-test-slave-lgr5"
server_key: "9d1fab92ba1ea7ab7154edcea103605a"
server_key_iv: "0aa96f0f48e2b3af532290b6b9cb69c3"
server_key_version: "encryptionkey@0"
I20260811 12:38:26.362785 4090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:26.394246 4090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:26.398645 4090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:26.398942 4090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:26.399982 4090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:26.401407 4090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:26.401489 4090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:26.401623 4090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:26.401678 4090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:26.432562 4090 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:37057
I20260811 12:38:26.432623 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:37057 every 8 connection(s)
I20260811 12:38:26.433897 4090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:38:26.439787 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 4090
I20260811 12:38:26.439970 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:38:26.440560 3889 external_mini_cluster.cc:1428] Setting key b73581b890348d815b7ec7e48b294a70
Aug 11 12:38:26 dist-test-slave-lgr5 krb5kdc[3603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451906, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:26.455494 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:26.436414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:59089 (local address 127.3.204.126:43459)
0811 12:38:26.436920 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:26.436925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:26.438121 (+ 1196us) server_negotiation.cc:407] Connection header received
0811 12:38:26.440131 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:26.440138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:26.440236 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:26.440360 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:26.442175 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:26.443094 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:26.444737 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:26.445306 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:26.445456 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:26.449031 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:26.449038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:26.449042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:26.449084 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:26.452189 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:26.452974 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:26.452980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:26.452983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:26.453143 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:26.453581 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:26.453585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:26.453588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:26.453932 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:26.454095 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:26.454891 (+ 796us) server_negotiation.cc:299] Negotiation successful
0811 12:38:26.455129 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260811 12:38:26.456409 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:26.437033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43459 (local address 127.3.204.66:59089)
0811 12:38:26.437940 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:26.437992 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:26.439735 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:26.440580 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:26.440589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:26.441216 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:26.442007 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:26.442034 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:26.443277 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:26.443281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:26.444593 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:26.444606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:26.445566 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:26.445572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:26.445773 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:26.446459 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:26.446485 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:26.448754 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:26.452347 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:26.452356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:26.452372 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:26.452845 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:26.453248 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:26.453252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:26.453254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:26.453479 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:26.454042 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:26.454048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:26.454400 (+ 352us) client_negotiation.cc:769] Sending connection context
0811 12:38:26.454758 (+ 358us) client_negotiation.cc:240] Negotiation successful
0811 12:38:26.455205 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":390,"threads_started":1}
I20260811 12:38:26.457904 4309 heartbeater.cc:347] Connected to a master server at 127.3.204.126:43459
I20260811 12:38:26.458212 4309 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:26.458971 4309 heartbeater.cc:511] Master 127.3.204.126:43459 requested a full tablet report, sending...
I20260811 12:38:26.460544 3701 ts_manager.cc:194] Registered new tserver with Master: 1e6c9b35ae5944919c1ba29aaa862880 (127.3.204.66:37057)
I20260811 12:38:26.461378 3701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:59089
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:38:26.508884 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43459
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:26.851573 4321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:26.851868 4321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:26.852017 4321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:26.859331 4321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:26.859462 4321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:26.859603 4321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:38:26.869768 4321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:43459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:26.871110 4321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:26.872503 4321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:26.886265 4330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:26.886482 4327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:26.886988 4321 server_base.cc:1034] running on GCE node
W20260811 12:38:26.886521 4328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:26.887887 4321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:26.889003 4321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:26.890213 4321 hybrid_clock.cc:648] HybridClock initialized: now 1786451906890160 us; error 34 us; skew 500 ppm
I20260811 12:38:26.890511 4321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:26 dist-test-slave-lgr5 krb5kdc[3603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451906, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:26.893554 4321 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:38:26.895079 4321 webserver.cc:460] Webserver started at http://127.3.204.67:34355/ using document root <none> and password file <none>
I20260811 12:38:26.895773 4321 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:26.895893 4321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:26.896265 4321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:26.899358 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "9590e01619c9497991946eb96fe96c4d"
format_stamp: "Formatted at 2026-08-11 12:38:26 on dist-test-slave-lgr5"
server_key: "b141afb05bae9411dfa8c156f6277944"
server_key_iv: "895681bb1ef448ccc8663448ff9d4401"
server_key_version: "encryptionkey@0"
I20260811 12:38:26.900180 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "9590e01619c9497991946eb96fe96c4d"
format_stamp: "Formatted at 2026-08-11 12:38:26 on dist-test-slave-lgr5"
server_key: "b141afb05bae9411dfa8c156f6277944"
server_key_iv: "895681bb1ef448ccc8663448ff9d4401"
server_key_version: "encryptionkey@0"
I20260811 12:38:26.905382 4321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 12:38:26.909060 4341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:26.910415 4321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:26.910575 4321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "9590e01619c9497991946eb96fe96c4d"
format_stamp: "Formatted at 2026-08-11 12:38:26 on dist-test-slave-lgr5"
server_key: "b141afb05bae9411dfa8c156f6277944"
server_key_iv: "895681bb1ef448ccc8663448ff9d4401"
server_key_version: "encryptionkey@0"
I20260811 12:38:26.910769 4321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:26.917428 4081 heartbeater.cc:503] Master 127.3.204.126:43459 was elected leader, sending a full tablet report...
I20260811 12:38:26.934031 4321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:26.938390 4321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:26.938675 4321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:26.939706 4321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:26.941136 4321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:26.941217 4321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:26.941341 4321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:26.941394 4321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:26.972904 4321 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:43327
I20260811 12:38:26.972977 4563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:43327 every 8 connection(s)
I20260811 12:38:26.974193 4321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:38:26.979148 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 4321
I20260811 12:38:26.979332 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:38:26.979859 3889 external_mini_cluster.cc:1428] Setting key 9b6b859a7184be3bf582eb7cdc0d536e
Aug 11 12:38:26 dist-test-slave-lgr5 krb5kdc[3603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451906, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:26.994273 4569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:26.976655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:40877 (local address 127.3.204.126:43459)
0811 12:38:26.977118 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:26.977124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:26.978110 (+ 986us) server_negotiation.cc:407] Connection header received
0811 12:38:26.980231 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:26.980237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:26.980336 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:26.980436 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:26.982271 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:26.983183 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:26.984819 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:26.985247 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:26.985360 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:26.988687 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:26.988692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:26.988695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:26.988728 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:26.991185 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:26.991902 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:26.991906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:26.991908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:26.992036 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:26.992493 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:26.992497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:26.992499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:26.992776 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:26.992928 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:26.993654 (+ 726us) server_negotiation.cc:299] Negotiation successful
0811 12:38:26.993895 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":204,"threads_started":1}
I20260811 12:38:26.995215 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:26.977209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:43459 (local address 127.3.204.67:40877)
0811 12:38:26.977939 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:26.977988 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:26.979896 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:26.980712 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:26.980721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:26.981331 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:26.982100 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:26.982121 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:26.983337 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:26.983341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:26.984674 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:26.984687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:26.985485 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:26.985493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:26.985723 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:26.986333 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:26.986356 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:26.988490 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:26.991318 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:26.991327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:26.991346 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:26.991778 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:26.992173 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:26.992178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:26.992182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:26.992374 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:26.992915 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:26.992923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:26.993259 (+ 336us) client_negotiation.cc:769] Sending connection context
0811 12:38:26.993506 (+ 247us) client_negotiation.cc:240] Negotiation successful
0811 12:38:26.994049 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":281,"threads_started":1}
I20260811 12:38:26.996462 4565 heartbeater.cc:347] Connected to a master server at 127.3.204.126:43459
I20260811 12:38:26.996747 4565 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:26.997440 4565 heartbeater.cc:511] Master 127.3.204.126:43459 requested a full tablet report, sending...
I20260811 12:38:26.998934 3701 ts_manager.cc:194] Registered new tserver with Master: 9590e01619c9497991946eb96fe96c4d (127.3.204.67:43327)
I20260811 12:38:26.999785 3701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:40877
I20260811 12:38:27.008123 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:38:27.025985 4569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.012376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.3.204.126:43459)
0811 12:38:27.012566 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.012572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.012972 (+ 400us) server_negotiation.cc:407] Connection header received
0811 12:38:27.013154 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.013160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.013232 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.013319 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:27.015011 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.015839 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.017219 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.017651 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.017752 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.020491 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:27.020497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:27.020500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:27.020535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:27.023334 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:27.024225 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:27.024229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:27.024232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:27.024362 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:27.024731 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:27.024734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:27.024736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:27.025061 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:27.025269 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.025605 (+ 336us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.025755 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 12:38:27.030035 3701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:38:27.037261 3701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:38:27.072402 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.051459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.3.204.65:41985)
0811 12:38:27.063818 (+ 12359us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.063827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.064611 (+ 784us) server_negotiation.cc:407] Connection header received
0811 12:38:27.064731 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.064740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.064989 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.065180 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:27.066051 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.067420 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.070102 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.071046 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.071177 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.071435 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.071832 (+ 397us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.072022 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12187,"thread_start_us":12039,"threads_started":1}
I20260811 12:38:27.072530 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.050472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:37057 (local address 127.0.0.1:45978)
0811 12:38:27.062376 (+ 11904us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.062406 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:27.062562 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:27.064372 (+ 1810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:27.064381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:27.064424 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:27.064921 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.064936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.067013 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.067018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.069421 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.069438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.071597 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.071603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.071816 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.071851 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:38:27.071927 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 12:38:27.072062 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11757,"mutex_wait_us":93,"thread_start_us":10471,"threads_started":1}
I20260811 12:38:27.072732 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.051137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.3.204.66:37057)
0811 12:38:27.063302 (+ 12165us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.063311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.063346 (+ 35us) server_negotiation.cc:407] Connection header received
0811 12:38:27.063470 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.063481 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.063771 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.064023 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:27.065148 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.066796 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.069588 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.071063 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.071198 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.071411 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.072234 (+ 823us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.072398 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12019,"thread_start_us":11907,"threads_started":1}
I20260811 12:38:27.074036 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.050860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:59990)
0811 12:38:27.064306 (+ 13446us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.064329 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:27.064472 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:27.065423 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:27.065428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:27.065464 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:27.065855 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.065867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.067559 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.067569 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.069935 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.069948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.071255 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.071262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.071483 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.071539 (+ 56us) client_negotiation.cc:769] Sending connection context
0811 12:38:27.071675 (+ 136us) client_negotiation.cc:240] Negotiation successful
0811 12:38:27.071876 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13314,"mutex_wait_us":3,"spinlock_wait_cycles":17664,"thread_start_us":13131,"threads_started":1}
I20260811 12:38:27.075299 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.050093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43327 (local address 127.0.0.1:37392)
0811 12:38:27.061517 (+ 11424us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.061583 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:27.061850 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:27.066809 (+ 4959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:27.066816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:27.066841 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:27.067839 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.067858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.069857 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.069863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.073112 (+ 3249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.073126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.074504 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.074512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.074673 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.074703 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 12:38:27.074800 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 12:38:27.074928 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11040,"mutex_wait_us":1223,"thread_start_us":12558,"threads_started":1}
I20260811 12:38:27.075392 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.050437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.3.204.67:43327)
0811 12:38:27.066256 (+ 15819us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.066264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.066300 (+ 36us) server_negotiation.cc:407] Connection header received
0811 12:38:27.066384 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.066392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.066571 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.066737 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:27.068073 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.069698 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.073281 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.074284 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.074440 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.074668 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.074849 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.075038 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15667,"thread_start_us":15495,"threads_started":1}
I20260811 12:38:27.076975 4192 tablet_service.cc:1467] Processing CreateTablet for tablet 39a4133cc49b410e9696f2587cba1f72 (DEFAULT_TABLE table=test-table [id=81257e535c9d4abba62ec4cd7af2485c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:27.078477 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a4133cc49b410e9696f2587cba1f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:27.077957 3962 tablet_service.cc:1467] Processing CreateTablet for tablet 39a4133cc49b410e9696f2587cba1f72 (DEFAULT_TABLE table=test-table [id=81257e535c9d4abba62ec4cd7af2485c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:27.079443 3962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a4133cc49b410e9696f2587cba1f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:27.079066 4432 tablet_service.cc:1467] Processing CreateTablet for tablet 39a4133cc49b410e9696f2587cba1f72 (DEFAULT_TABLE table=test-table [id=81257e535c9d4abba62ec4cd7af2485c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:27.080607 4432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a4133cc49b410e9696f2587cba1f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:27.090775 4606 tablet_bootstrap.cc:492] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Bootstrap starting.
I20260811 12:38:27.094062 4606 tablet_bootstrap.cc:654] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:27.094964 4610 tablet_bootstrap.cc:492] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: Bootstrap starting.
I20260811 12:38:27.095933 4606 log.cc:824] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:27.099229 4610 tablet_bootstrap.cc:654] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:27.100242 4614 tablet_bootstrap.cc:492] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Bootstrap starting.
I20260811 12:38:27.100387 4606 tablet_bootstrap.cc:492] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: No bootstrap required, opened a new log
I20260811 12:38:27.100636 4610 log.cc:824] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:27.100773 4606 ts_tablet_manager.cc:1397] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260811 12:38:27.103566 4614 tablet_bootstrap.cc:654] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:27.103904 4610 tablet_bootstrap.cc:492] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: No bootstrap required, opened a new log
I20260811 12:38:27.104228 4610 ts_tablet_manager.cc:1397] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260811 12:38:27.104741 4614 log.cc:824] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:27.110628 4614 tablet_bootstrap.cc:492] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: No bootstrap required, opened a new log
I20260811 12:38:27.110926 4614 ts_tablet_manager.cc:1397] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260811 12:38:27.111202 4606 raft_consensus.cc:348] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.111706 4606 raft_consensus.cc:374] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:27.111820 4606 raft_consensus.cc:729] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6c9b35ae5944919c1ba29aaa862880, State: Initialized, Role: FOLLOWER
I20260811 12:38:27.112644 4606 consensus_queue.cc:260] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.113915 4309 heartbeater.cc:503] Master 127.3.204.126:43459 was elected leader, sending a full tablet report...
I20260811 12:38:27.114706 4606 ts_tablet_manager.cc:1428] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260811 12:38:27.114653 4610 raft_consensus.cc:348] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.115136 4610 raft_consensus.cc:374] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:27.115294 4610 raft_consensus.cc:729] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92225c6dc01d4a3090358db670a12d87, State: Initialized, Role: FOLLOWER
I20260811 12:38:27.115953 4610 consensus_queue.cc:260] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.118114 4610 ts_tablet_manager.cc:1428] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260811 12:38:27.120530 4614 raft_consensus.cc:348] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.121008 4614 raft_consensus.cc:374] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:27.121124 4614 raft_consensus.cc:729] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9590e01619c9497991946eb96fe96c4d, State: Initialized, Role: FOLLOWER
I20260811 12:38:27.121702 4614 consensus_queue.cc:260] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.122759 4565 heartbeater.cc:503] Master 127.3.204.126:43459 was elected leader, sending a full tablet report...
I20260811 12:38:27.123257 4614 ts_tablet_manager.cc:1428] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
W20260811 12:38:27.136420 4083 tablet.cc:2369] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:38:27.153488 4627 raft_consensus.cc:482] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:27.153831 4627 raft_consensus.cc:504] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.156914 4627 leader_election.cc:288] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9590e01619c9497991946eb96fe96c4d (127.3.204.67:43327), 1e6c9b35ae5944919c1ba29aaa862880 (127.3.204.66:37057)
I20260811 12:38:27.162791 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.156321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:36983 (local address 127.3.204.66:37057)
0811 12:38:27.156504 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.156509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.157156 (+ 647us) server_negotiation.cc:407] Connection header received
0811 12:38:27.157356 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.157362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.157442 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.157510 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:27.158106 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.158870 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.160845 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.161783 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.161904 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.161960 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.162370 (+ 410us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.162509 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 12:38:27.162864 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.156106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:37057 (local address 127.3.204.65:36983)
0811 12:38:27.157010 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.157030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:27.157201 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:27.157550 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:27.157554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:27.157581 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:27.157947 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.157962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.159001 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.159006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.160668 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.160685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.162007 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.162013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.162194 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.162227 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 12:38:27.162312 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 12:38:27.162416 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":1,"thread_start_us":510,"threads_started":1}
I20260811 12:38:27.162919 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.156202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:57347 (local address 127.3.204.67:43327)
0811 12:38:27.156434 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.156441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.157178 (+ 737us) server_negotiation.cc:407] Connection header received
0811 12:38:27.157437 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.157442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.157516 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.157608 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:27.158287 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.159073 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.161175 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.161997 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.162152 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.162223 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.162558 (+ 335us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.162692 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260811 12:38:27.163053 4634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.156145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43327 (local address 127.3.204.65:57347)
0811 12:38:27.157068 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.157085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:27.157284 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:27.157639 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:27.157646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:27.157738 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:27.158127 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.158139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.159233 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.159237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.161021 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:27.161037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.162214 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.162219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:27.162389 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.162421 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 12:38:27.162511 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 12:38:27.162640 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":37,"spinlock_wait_cycles":19328,"thread_start_us":534,"threads_started":1}
I20260811 12:38:27.163969 4228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a4133cc49b410e9696f2587cba1f72" candidate_uuid: "92225c6dc01d4a3090358db670a12d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6c9b35ae5944919c1ba29aaa862880" is_pre_election: true
I20260811 12:38:27.163981 4476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a4133cc49b410e9696f2587cba1f72" candidate_uuid: "92225c6dc01d4a3090358db670a12d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9590e01619c9497991946eb96fe96c4d" is_pre_election: true
I20260811 12:38:27.164511 4476 raft_consensus.cc:2454] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92225c6dc01d4a3090358db670a12d87 in term 0.
I20260811 12:38:27.164511 4228 raft_consensus.cc:2454] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92225c6dc01d4a3090358db670a12d87 in term 0.
I20260811 12:38:27.165215 3859 leader_election.cc:302] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e6c9b35ae5944919c1ba29aaa862880, 92225c6dc01d4a3090358db670a12d87, 9590e01619c9497991946eb96fe96c4d; no voters:
I20260811 12:38:27.165684 4627 raft_consensus.cc:2781] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:38:27.165825 4627 raft_consensus.cc:482] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:38:27.165926 4627 raft_consensus.cc:3037] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:27.168569 4627 raft_consensus.cc:504] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.169258 4627 leader_election.cc:288] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [CANDIDATE]: Term 1 election: Requested vote from peers 9590e01619c9497991946eb96fe96c4d (127.3.204.67:43327), 1e6c9b35ae5944919c1ba29aaa862880 (127.3.204.66:37057)
I20260811 12:38:27.169759 4476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a4133cc49b410e9696f2587cba1f72" candidate_uuid: "92225c6dc01d4a3090358db670a12d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9590e01619c9497991946eb96fe96c4d"
I20260811 12:38:27.169796 4228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a4133cc49b410e9696f2587cba1f72" candidate_uuid: "92225c6dc01d4a3090358db670a12d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6c9b35ae5944919c1ba29aaa862880"
I20260811 12:38:27.169951 4476 raft_consensus.cc:3037] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:27.170027 4228 raft_consensus.cc:3037] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:27.172310 4476 raft_consensus.cc:2454] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92225c6dc01d4a3090358db670a12d87 in term 1.
I20260811 12:38:27.172379 4228 raft_consensus.cc:2454] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92225c6dc01d4a3090358db670a12d87 in term 1.
I20260811 12:38:27.172745 3859 leader_election.cc:302] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92225c6dc01d4a3090358db670a12d87, 9590e01619c9497991946eb96fe96c4d; no voters:
I20260811 12:38:27.173092 4627 raft_consensus.cc:2781] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:27.173542 4627 raft_consensus.cc:686] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 1 LEADER]: Becoming Leader. State: Replica: 92225c6dc01d4a3090358db670a12d87, State: Running, Role: LEADER
I20260811 12:38:27.174019 4627 consensus_queue.cc:237] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } }
I20260811 12:38:27.178918 3698 catalog_manager.cc:5495] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92225c6dc01d4a3090358db670a12d87 (127.3.204.65). New cstate: current_term: 1 leader_uuid: "92225c6dc01d4a3090358db670a12d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92225c6dc01d4a3090358db670a12d87" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 41985 } health_report { overall_health: HEALTHY } } }
W20260811 12:38:27.185822 4310 tablet.cc:2369] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:38:27.205588 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.200038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.3.204.65:41985)
0811 12:38:27.200279 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.200285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.200303 (+ 18us) server_negotiation.cc:407] Connection header received
0811 12:38:27.200395 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.200399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.200522 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.200658 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:38:27.201077 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.201785 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.203213 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.203639 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.203749 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.204431 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:38:27.204971 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:38:27.205138 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.205298 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.205349 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 12:38:27.214112 4228 raft_consensus.cc:1261] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 1 FOLLOWER]: Refusing update from remote peer 92225c6dc01d4a3090358db670a12d87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:27.214112 4476 raft_consensus.cc:1261] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 1 FOLLOWER]: Refusing update from remote peer 92225c6dc01d4a3090358db670a12d87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:27.215178 4636 consensus_queue.cc:1035] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9590e01619c9497991946eb96fe96c4d" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:27.215592 4627 consensus_queue.cc:1035] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e6c9b35ae5944919c1ba29aaa862880" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 12:38:27.225816 4566 tablet.cc:2369] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:38:27.228439 4644 mvcc.cc:205] Tried to move back new op lower bound from 7317307011938635776 to 7317307011788525568. Current Snapshot: MvccSnapshot[applied={T|T < 7317307011938635776}]
I20260811 12:38:27.229067 4647 mvcc.cc:205] Tried to move back new op lower bound from 7317307011938635776 to 7317307011788525568. Current Snapshot: MvccSnapshot[applied={T|T < 7317307011938635776}]
I20260811 12:38:27.242393 3701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:27.242667 3701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:27.246217 3701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36892538e4084b72881f73c4bb83d096: Sending DeleteTablet for 3 replicas of tablet 39a4133cc49b410e9696f2587cba1f72
I20260811 12:38:27.247419 4192 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a4133cc49b410e9696f2587cba1f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:27 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:45978
I20260811 12:38:27.247413 4432 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a4133cc49b410e9696f2587cba1f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:27 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:37392
I20260811 12:38:27.251873 3962 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a4133cc49b410e9696f2587cba1f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:27 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:59990
I20260811 12:38:27.254359 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 3818
I20260811 12:38:27.266152 4666 tablet_replica.cc:330] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: stopping tablet replica
I20260811 12:38:27.267374 4666 raft_consensus.cc:2229] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:38:27.268911 4666 raft_consensus.cc:2258] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:27.272723 4666 ts_tablet_manager.cc:1905] T 39a4133cc49b410e9696f2587cba1f72 P 92225c6dc01d4a3090358db670a12d87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:27.274457 4664 tablet_replica.cc:330] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: stopping tablet replica
I20260811 12:38:27.275100 4663 tablet_replica.cc:330] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: stopping tablet replica
I20260811 12:38:27.275590 4663 raft_consensus.cc:2229] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:27.276190 4663 raft_consensus.cc:2258] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 11 12:38:27 dist-test-slave-lgr5 krb5kdc[3603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:38:27.278949 4663 ts_tablet_manager.cc:1905] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:27.280153 4664 raft_consensus.cc:2229] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:27.280865 4664 raft_consensus.cc:2258] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:27.283864 4664 ts_tablet_manager.cc:1905] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:27.285840 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.255805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.3.204.65:41985)
0811 12:38:27.256019 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.256026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.265128 (+ 9102us) server_negotiation.cc:407] Connection header received
0811 12:38:27.265332 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.265338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.265452 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.265567 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:27.267871 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.269320 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.271371 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.272136 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.272247 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.277689 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:27.277708 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:27.277727 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:27.277783 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:27.281934 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:27.282908 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:27.282919 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:27.282923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:27.283118 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:27.283662 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:27.283667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:27.283669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:27.284223 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:27.284542 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.285190 (+ 648us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.285479 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:38:27.287248 4663 ts_tablet_manager.cc:1918] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:27.287387 4066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:27.287508 4663 log.cc:1196] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/39a4133cc49b410e9696f2587cba1f72
I20260811 12:38:27.288240 4663 ts_tablet_manager.cc:1939] T 39a4133cc49b410e9696f2587cba1f72 P 1e6c9b35ae5944919c1ba29aaa862880: Deleting consensus metadata
I20260811 12:38:27.291072 3663 catalog_manager.cc:4841] TS 1e6c9b35ae5944919c1ba29aaa862880 (127.3.204.66:37057): tablet 39a4133cc49b410e9696f2587cba1f72 (table test-table [id=81257e535c9d4abba62ec4cd7af2485c]) successfully deleted
I20260811 12:38:27.293308 4664 ts_tablet_manager.cc:1918] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:27.293493 4664 log.cc:1196] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/39a4133cc49b410e9696f2587cba1f72
I20260811 12:38:27.294071 4664 ts_tablet_manager.cc:1939] T 39a4133cc49b410e9696f2587cba1f72 P 9590e01619c9497991946eb96fe96c4d: Deleting consensus metadata
I20260811 12:38:27.296114 3667 catalog_manager.cc:4841] TS 9590e01619c9497991946eb96fe96c4d (127.3.204.67:43327): tablet 39a4133cc49b410e9696f2587cba1f72 (table test-table [id=81257e535c9d4abba62ec4cd7af2485c]) successfully deleted
I20260811 12:38:27.389535 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 3818
I20260811 12:38:27.413506 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 4090
W20260811 12:38:27.413421 3666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.65:41985 (error 108)
I20260811 12:38:27.415306 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.414848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:60018)
0811 12:38:27.415045 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.415163 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 12:38:27.415685 3666 catalog_manager.cc:4584] TS 92225c6dc01d4a3090358db670a12d87 (127.3.204.65:41985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39a4133cc49b410e9696f2587cba1f72: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Aug 11 12:38:27 dist-test-slave-lgr5 krb5kdc[3603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:38:27.432345 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.414369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.3.204.66:37057)
0811 12:38:27.414561 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.414567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.414587 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:38:27.414769 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.414775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.414841 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.414943 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:27.416662 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.417396 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.418616 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.418994 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.419083 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.426564 (+ 7481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:27.426583 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:27.426600 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:27.426648 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:27.429415 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:27.430152 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:27.430161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:27.430163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:27.430312 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:27.430648 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:27.430651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:27.430653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:27.431059 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:27.431296 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.431669 (+ 373us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.431894 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:38:27.433202 4301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:27.469542 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.468917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:60032)
0811 12:38:27.469186 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.469319 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260811 12:38:27.470602 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.470235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:60036)
0811 12:38:27.470383 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.470481 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 12:38:27.522500 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 4090
I20260811 12:38:27.531672 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.531040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:60038)
0811 12:38:27.531295 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.531434 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260811 12:38:27.532724 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.532408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:60040)
0811 12:38:27.532525 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.532592 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260811 12:38:27.549662 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 4321
Aug 11 12:38:27 dist-test-slave-lgr5 krb5kdc[3603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:38:27.564996 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.550419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.3.204.67:43327)
0811 12:38:27.550634 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:27.550639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:27.550659 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:38:27.550751 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:27.550756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:27.550832 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:27.550915 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:27.552444 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.553210 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.554582 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:27.555023 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:27.555125 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:27.559643 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:27.559661 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:27.559679 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:27.559720 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:27.562480 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:27.563097 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:27.563105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:27.563107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:27.563250 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:27.563538 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:27.563541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:27.563543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:27.563904 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:27.564176 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:27.564500 (+ 324us) server_negotiation.cc:299] Negotiation successful
0811 12:38:27.564681 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260811 12:38:27.565688 4558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:27.613673 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.613107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:60054)
0811 12:38:27.613380 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.613519 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260811 12:38:27.614765 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.614373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:60064)
0811 12:38:27.614578 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.614650 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260811 12:38:27.655396 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 4321
I20260811 12:38:27.683205 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 3636
I20260811 12:38:27.684563 3754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:27.795625 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:27.794881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:41985 (local address 127.0.0.1:60070)
0811 12:38:27.795126 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:27.795273 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 12:38:27.795840 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 3636
2026-08-11T12:38:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4622 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-11T12:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:27Z Disabled control of system clock
I20260811 12:38:27.891881 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33257
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:33257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 12:38:28.240671 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:28.240988 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:28.241077 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:28.248385 4679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:28.248502 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:28.248545 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:28.248601 4679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:28.248644 4679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:28.258030 4679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:33257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:28.259332 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:28.261260 4679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:28.273092 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:28.273164 4689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:28.273203 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:28.273739 4679 server_base.cc:1034] running on GCE node
I20260811 12:38:28.274631 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:28.276141 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:28.277374 4679 hybrid_clock.cc:648] HybridClock initialized: now 1786451908277316 us; error 47 us; skew 500 ppm
I20260811 12:38:28.277711 4679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:28.280110 4679 webserver.cc:460] Webserver started at http://127.3.204.126:38473/ using document root <none> and password file <none>
I20260811 12:38:28.280910 4679 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:28.281005 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:28.281343 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:28.284461 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "4635655b82c44aab84ec7683de251c93"
format_stamp: "Formatted at 2026-08-11 12:38:28 on dist-test-slave-lgr5"
server_key: "49aa8fd28a03fcc31e522e3b8b6f88cc"
server_key_iv: "7bd953b7effa45fc4ab0f72f7ac0aa9d"
server_key_version: "encryptionkey@0"
I20260811 12:38:28.285315 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "4635655b82c44aab84ec7683de251c93"
format_stamp: "Formatted at 2026-08-11 12:38:28 on dist-test-slave-lgr5"
server_key: "49aa8fd28a03fcc31e522e3b8b6f88cc"
server_key_iv: "7bd953b7effa45fc4ab0f72f7ac0aa9d"
server_key_version: "encryptionkey@0"
I20260811 12:38:28.290807 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 12:38:28.294494 4703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:28.295838 4679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 12:38:28.296032 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "4635655b82c44aab84ec7683de251c93"
format_stamp: "Formatted at 2026-08-11 12:38:28 on dist-test-slave-lgr5"
server_key: "49aa8fd28a03fcc31e522e3b8b6f88cc"
server_key_iv: "7bd953b7effa45fc4ab0f72f7ac0aa9d"
server_key_version: "encryptionkey@0"
I20260811 12:38:28.296311 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:28.318164 4679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:28.319074 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:28.319358 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:28.337217 4679 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:33257
I20260811 12:38:28.337255 4800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:33257 every 8 connection(s)
I20260811 12:38:28.338601 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:28.341552 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 4679
I20260811 12:38:28.341715 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:28.342164 3889 external_mini_cluster.cc:1428] Setting key 6380a5f8a029d6e934780411a145a2e6
I20260811 12:38:28.342617 4803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:28.355290 4803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93: Bootstrap starting.
I20260811 12:38:28.355881 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:28.343962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.3.204.126:33257)
0811 12:38:28.344976 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:28.344987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:28.345028 (+ 41us) server_negotiation.cc:407] Connection header received
0811 12:38:28.347301 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:28.347328 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:28.347794 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:28.348192 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:28.348771 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:28.350085 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:28.351443 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:28.352180 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:28.352349 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:28.353552 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:28.353561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:28.353576 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:28.354116 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:28.354194 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:28.354216 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:28.354323 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:28.354591 (+ 268us) server_negotiation.cc:299] Negotiation successful
0811 12:38:28.354915 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":349,"threads_started":1}
I20260811 12:38:28.358935 4803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:28.359998 4803 log.cc:824] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:28.362857 4803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93: No bootstrap required, opened a new log
I20260811 12:38:28.370365 4803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4635655b82c44aab84ec7683de251c93" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33257 } }
I20260811 12:38:28.370784 4803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:28.370867 4803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4635655b82c44aab84ec7683de251c93, State: Initialized, Role: FOLLOWER
I20260811 12:38:28.371415 4803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4635655b82c44aab84ec7683de251c93" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33257 } }
I20260811 12:38:28.371640 4803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:28.371743 4803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:28.371908 4803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:28.374385 4803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4635655b82c44aab84ec7683de251c93" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33257 } }
I20260811 12:38:28.374873 4803 leader_election.cc:302] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4635655b82c44aab84ec7683de251c93; no voters:
I20260811 12:38:28.375463 4803 leader_election.cc:288] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:28.375645 4816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:28.376410 4816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [term 1 LEADER]: Becoming Leader. State: Replica: 4635655b82c44aab84ec7683de251c93, State: Running, Role: LEADER
I20260811 12:38:28.377017 4816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4635655b82c44aab84ec7683de251c93" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33257 } }
I20260811 12:38:28.377581 4803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:28.380192 4818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4635655b82c44aab84ec7683de251c93. Latest consensus state: current_term: 1 leader_uuid: "4635655b82c44aab84ec7683de251c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4635655b82c44aab84ec7683de251c93" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33257 } } }
I20260811 12:38:28.380481 4817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4635655b82c44aab84ec7683de251c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4635655b82c44aab84ec7683de251c93" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33257 } } }
I20260811 12:38:28.380821 4818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:28.380818 4817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:28.382661 4828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:28.388355 4828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:28.397353 4828 catalog_manager.cc:1347] Generated new cluster ID: 349787f50c9a4e8ab0ac44c5fb1a29e4
I20260811 12:38:28.397483 4828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:28.410233 4828 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:28.412184 4828 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:28.421011 4828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93: Generated new TSK 0
I20260811 12:38:28.422215 4828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:28.435772 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33257
--builtin_ntp_servers=127.3.204.84:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 12:38:28.783322 4853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:28.783624 4853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:28.783748 4853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:28.791267 4853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:28.791383 4853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:28.791536 4853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:38:28.801323 4853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:28.802663 4853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:28.804507 4853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:28.817320 4867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:28.817206 4868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:28.817216 4874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:28.817998 4853 server_base.cc:1034] running on GCE node
I20260811 12:38:28.818590 4853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:28.819504 4853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:28.820703 4853 hybrid_clock.cc:648] HybridClock initialized: now 1786451908820657 us; error 34 us; skew 500 ppm
I20260811 12:38:28.821007 4853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:28.823355 4853 webserver.cc:460] Webserver started at http://127.3.204.65:42519/ using document root <none> and password file <none>
I20260811 12:38:28.824024 4853 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:28.824150 4853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:28.824447 4853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:28.827277 4853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "3243b7a2c11342a7983a3b1aca49ac2b"
format_stamp: "Formatted at 2026-08-11 12:38:28 on dist-test-slave-lgr5"
server_key: "f6ba5df8e5dd52e8807f38e4b5d085bd"
server_key_iv: "cd16ac0b0692b8f50a45de346568d943"
server_key_version: "encryptionkey@0"
I20260811 12:38:28.828068 4853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "3243b7a2c11342a7983a3b1aca49ac2b"
format_stamp: "Formatted at 2026-08-11 12:38:28 on dist-test-slave-lgr5"
server_key: "f6ba5df8e5dd52e8807f38e4b5d085bd"
server_key_iv: "cd16ac0b0692b8f50a45de346568d943"
server_key_version: "encryptionkey@0"
I20260811 12:38:28.833194 4853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 12:38:28.836715 4886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:28.838084 4853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:28.838238 4853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "3243b7a2c11342a7983a3b1aca49ac2b"
format_stamp: "Formatted at 2026-08-11 12:38:28 on dist-test-slave-lgr5"
server_key: "f6ba5df8e5dd52e8807f38e4b5d085bd"
server_key_iv: "cd16ac0b0692b8f50a45de346568d943"
server_key_version: "encryptionkey@0"
I20260811 12:38:28.838434 4853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:28.862752 4853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:28.863616 4853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:28.863902 4853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:28.864931 4853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:28.866405 4853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:28.866500 4853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:28.866581 4853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:28.866641 4853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:28.897367 4853 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:46613
I20260811 12:38:28.897444 5088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:46613 every 8 connection(s)
I20260811 12:38:28.898727 4853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:38:28.899019 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 4853
I20260811 12:38:28.899179 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:38:28.899793 3889 external_mini_cluster.cc:1428] Setting key dc9077d2cff778c2aa5512ce9ffaaf97
I20260811 12:38:28.903174 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33257
--builtin_ntp_servers=127.3.204.84:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 12:38:28.911787 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:28.901562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:52159 (local address 127.3.204.126:33257)
0811 12:38:28.902139 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:28.902144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:28.903510 (+ 1366us) server_negotiation.cc:407] Connection header received
0811 12:38:28.905407 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:28.905412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:28.905490 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:28.905584 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:28.906686 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:28.907443 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:28.909062 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:28.909469 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:28.909578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:28.910956 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:28.910961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:28.910964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:28.910984 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:28.911025 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:28.911037 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:28.911110 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:28.911332 (+ 222us) server_negotiation.cc:299] Negotiation successful
0811 12:38:28.911438 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":384,"threads_started":1}
I20260811 12:38:28.912847 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:28.902297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33257 (local address 127.3.204.65:52159)
0811 12:38:28.903359 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:28.903411 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:28.905142 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:28.905845 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:28.905853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:28.905950 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:28.906518 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:28.906535 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:28.907592 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:28.907596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:28.908920 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:28.908933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:28.909699 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:28.909706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:28.909952 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:28.910625 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:28.910628 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:28.910670 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:28.910759 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:28.911122 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:28.911134 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:28.911142 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 12:38:28.911301 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 12:38:28.911715 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":341,"threads_started":1}
I20260811 12:38:28.914455 5089 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33257
I20260811 12:38:28.914749 5089 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:28.915513 5089 heartbeater.cc:511] Master 127.3.204.126:33257 requested a full tablet report, sending...
I20260811 12:38:28.917654 4745 ts_manager.cc:194] Registered new tserver with Master: 3243b7a2c11342a7983a3b1aca49ac2b (127.3.204.65:46613)
I20260811 12:38:28.919201 4745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.65:52159
W20260811 12:38:29.247551 5094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:29.247870 5094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:29.247999 5094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:29.255375 5094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:29.255498 5094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:29.255653 5094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:38:29.265002 5094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:29.266361 5094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:29.268229 5094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:29.282096 5108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:29.282037 5113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:29.282075 5110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:29.282853 5094 server_base.cc:1034] running on GCE node
I20260811 12:38:29.283527 5094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:29.284564 5094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:29.285775 5094 hybrid_clock.cc:648] HybridClock initialized: now 1786451909285722 us; error 43 us; skew 500 ppm
I20260811 12:38:29.286099 5094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:29.288475 5094 webserver.cc:460] Webserver started at http://127.3.204.66:41763/ using document root <none> and password file <none>
I20260811 12:38:29.289209 5094 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:29.289311 5094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:29.289649 5094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:29.292922 5094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "766488a9bdd1416e8b7a06659e9a3dec"
format_stamp: "Formatted at 2026-08-11 12:38:29 on dist-test-slave-lgr5"
server_key: "5cce8303f4c4ad2a7c78ece2984d35ac"
server_key_iv: "702965910c56b29ed3f6df773e16b59d"
server_key_version: "encryptionkey@0"
I20260811 12:38:29.293763 5094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "766488a9bdd1416e8b7a06659e9a3dec"
format_stamp: "Formatted at 2026-08-11 12:38:29 on dist-test-slave-lgr5"
server_key: "5cce8303f4c4ad2a7c78ece2984d35ac"
server_key_iv: "702965910c56b29ed3f6df773e16b59d"
server_key_version: "encryptionkey@0"
I20260811 12:38:29.299279 5094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:38:29.303359 5122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:29.304901 5094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:38:29.305073 5094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "766488a9bdd1416e8b7a06659e9a3dec"
format_stamp: "Formatted at 2026-08-11 12:38:29 on dist-test-slave-lgr5"
server_key: "5cce8303f4c4ad2a7c78ece2984d35ac"
server_key_iv: "702965910c56b29ed3f6df773e16b59d"
server_key_version: "encryptionkey@0"
I20260811 12:38:29.305290 5094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:29.329876 5094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:29.330770 5094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:29.331084 5094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:29.332183 5094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:29.333756 5094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:29.333854 5094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:29.333966 5094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:29.334043 5094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:29.367108 5094 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:36303
I20260811 12:38:29.367167 5331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:36303 every 8 connection(s)
I20260811 12:38:29.368480 5094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:38:29.376052 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 5094
I20260811 12:38:29.376241 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:38:29.376679 3889 external_mini_cluster.cc:1428] Setting key 76e4a929deee87005652c6c8b2671f86
I20260811 12:38:29.379590 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33257
--builtin_ntp_servers=127.3.204.84:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 12:38:29.381304 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.371112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:58533 (local address 127.3.204.126:33257)
0811 12:38:29.371286 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:29.371291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:29.372728 (+ 1437us) server_negotiation.cc:407] Connection header received
0811 12:38:29.374509 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:29.374514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:29.374587 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:29.374677 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:29.375786 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.376623 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.378550 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.378971 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.379083 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.380504 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:29.380509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:29.380512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:29.380529 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:29.380561 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:29.380573 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:29.380645 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:29.380865 (+ 220us) server_negotiation.cc:299] Negotiation successful
0811 12:38:29.380976 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 12:38:29.382400 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.371593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33257 (local address 127.3.204.66:58533)
0811 12:38:29.372579 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:29.372617 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:29.374260 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:29.374869 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:29.374877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:29.375003 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:29.375594 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.375612 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.376790 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.376796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.378410 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.378427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.379201 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.379209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.379421 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.380178 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:29.380181 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:29.380232 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:29.380319 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:29.380661 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:29.380672 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:29.380681 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 12:38:29.380840 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 12:38:29.381246 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":290,"threads_started":1}
I20260811 12:38:29.383862 5332 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33257
I20260811 12:38:29.384186 5332 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:29.384917 5332 heartbeater.cc:511] Master 127.3.204.126:33257 requested a full tablet report, sending...
I20260811 12:38:29.386325 4745 ts_manager.cc:194] Registered new tserver with Master: 766488a9bdd1416e8b7a06659e9a3dec (127.3.204.66:36303)
I20260811 12:38:29.387117 4745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.66:58533
W20260811 12:38:29.722554 5337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:29.722862 5337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:29.722954 5337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:29.730336 5337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:29.730446 5337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:29.730561 5337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:38:29.740250 5337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:29.741606 5337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:29.743583 5337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:29.757140 5352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:29.757210 5358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:29.757179 5354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:29.757845 5337 server_base.cc:1034] running on GCE node
I20260811 12:38:29.758497 5337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:29.759474 5337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:29.760684 5337 hybrid_clock.cc:648] HybridClock initialized: now 1786451909760624 us; error 50 us; skew 500 ppm
I20260811 12:38:29.761022 5337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:29.763437 5337 webserver.cc:460] Webserver started at http://127.3.204.67:38871/ using document root <none> and password file <none>
I20260811 12:38:29.764212 5337 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:29.764326 5337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:29.764667 5337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:29.767748 5337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "a7d8af92656442048c4a41708a3f22df"
format_stamp: "Formatted at 2026-08-11 12:38:29 on dist-test-slave-lgr5"
server_key: "997469964f5f82a7d2129a2fabf1b3b0"
server_key_iv: "d4b23144ce076695cf39740c58b18a75"
server_key_version: "encryptionkey@0"
I20260811 12:38:29.768651 5337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "a7d8af92656442048c4a41708a3f22df"
format_stamp: "Formatted at 2026-08-11 12:38:29 on dist-test-slave-lgr5"
server_key: "997469964f5f82a7d2129a2fabf1b3b0"
server_key_iv: "d4b23144ce076695cf39740c58b18a75"
server_key_version: "encryptionkey@0"
I20260811 12:38:29.774559 5337 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260811 12:38:29.778241 5365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:29.779682 5337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:38:29.779853 5337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "a7d8af92656442048c4a41708a3f22df"
format_stamp: "Formatted at 2026-08-11 12:38:29 on dist-test-slave-lgr5"
server_key: "997469964f5f82a7d2129a2fabf1b3b0"
server_key_iv: "d4b23144ce076695cf39740c58b18a75"
server_key_version: "encryptionkey@0"
I20260811 12:38:29.780135 5337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:29.817909 5337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:29.818868 5337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:29.819183 5337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:29.820315 5337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:29.821949 5337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:29.822058 5337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:29.822171 5337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:29.822249 5337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:29.853820 5337 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:44309
I20260811 12:38:29.853878 5593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:44309 every 8 connection(s)
I20260811 12:38:29.855198 5337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:38:29.861832 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 5337
I20260811 12:38:29.861995 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:38:29.862454 3889 external_mini_cluster.cc:1428] Setting key b35e43bc6575a88df838b00581db999a
I20260811 12:38:29.868422 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.857817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:41473 (local address 127.3.204.126:33257)
0811 12:38:29.858012 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:29.858017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:29.859497 (+ 1480us) server_negotiation.cc:407] Connection header received
0811 12:38:29.861374 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:29.861380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:29.861473 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:29.861564 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:29.862846 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.863778 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.865732 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.866151 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.866263 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.867607 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:29.867612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:29.867615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:29.867634 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:29.867667 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:29.867679 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:29.867753 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:29.867973 (+ 220us) server_negotiation.cc:299] Negotiation successful
0811 12:38:29.868103 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:38:29.869516 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.858385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:33257 (local address 127.3.204.67:41473)
0811 12:38:29.859303 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:29.859349 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:29.861088 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:29.861824 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:29.861832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:29.861939 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:29.862669 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.862692 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.863932 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.863936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.865588 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.865603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.866372 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.866381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.866614 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.867280 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:29.867283 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:29.867328 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:29.867418 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:29.867767 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:29.867779 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:29.867787 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 12:38:29.867949 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 12:38:29.868429 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
I20260811 12:38:29.870922 5594 heartbeater.cc:347] Connected to a master server at 127.3.204.126:33257
I20260811 12:38:29.871236 5594 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:29.872046 5594 heartbeater.cc:511] Master 127.3.204.126:33257 requested a full tablet report, sending...
I20260811 12:38:29.873558 4745 ts_manager.cc:194] Registered new tserver with Master: a7d8af92656442048c4a41708a3f22df (127.3.204.67:44309)
I20260811 12:38:29.874347 4745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.67:41473
I20260811 12:38:29.877897 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:38:29.888265 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.882192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.3.204.126:33257)
0811 12:38:29.882443 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:29.882449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:29.882756 (+ 307us) server_negotiation.cc:407] Connection header received
0811 12:38:29.882909 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:29.882914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:29.883014 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:29.883136 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:29.883686 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.884458 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.885960 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.886375 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.886470 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.887554 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:29.887559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:29.887562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:29.887581 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:29.887615 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:29.887627 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:29.887739 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:29.887884 (+ 145us) server_negotiation.cc:299] Negotiation successful
0811 12:38:29.887938 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 12:38:29.892211 4745 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:38:29.899477 4745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:38:29.922828 5089 heartbeater.cc:503] Master 127.3.204.126:33257 was elected leader, sending a full tablet report...
I20260811 12:38:29.938306 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.912207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:44309 (local address 127.0.0.1:57326)
0811 12:38:29.928140 (+ 15933us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:29.928159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:29.928283 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:29.929225 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:29.929230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:29.929257 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:29.929685 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.929699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.931620 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.931626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.934930 (+ 3304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.934941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.937251 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.937259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.937476 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.937531 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 12:38:29.937666 (+ 135us) client_negotiation.cc:240] Negotiation successful
0811 12:38:29.937864 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15760,"mutex_wait_us":47,"thread_start_us":16066,"threads_started":1}
I20260811 12:38:29.939277 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.912400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.3.204.67:44309)
0811 12:38:29.927043 (+ 14643us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:29.927054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:29.928271 (+ 1217us) server_negotiation.cc:407] Connection header received
0811 12:38:29.928480 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:29.928488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:29.928729 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:29.928914 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:29.929886 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.931442 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.935172 (+ 3730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.936638 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.936766 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.937109 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:29.938700 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0811 12:38:29.938900 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14428,"thread_start_us":14226,"threads_started":1}
I20260811 12:38:29.939713 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.912762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.3.204.66:36303)
0811 12:38:29.927504 (+ 14742us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:29.927513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:29.927691 (+ 178us) server_negotiation.cc:407] Connection header received
0811 12:38:29.928138 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:29.928146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:29.928471 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:29.929554 (+ 1083us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:29.931165 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.932685 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.934836 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.937811 (+ 2975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.938894 (+ 1083us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.939140 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:29.939208 (+ 68us) server_negotiation.cc:299] Negotiation successful
0811 12:38:29.939381 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14562,"thread_start_us":14751,"threads_started":1}
I20260811 12:38:29.938760 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.912200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:36303 (local address 127.0.0.1:56222)
0811 12:38:29.927473 (+ 15273us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:29.927529 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:29.927764 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:29.928730 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:29.928737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:29.928767 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:29.929297 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.929315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.932849 (+ 3534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.932854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.934638 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.934653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.938021 (+ 3368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.938028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.938207 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.938242 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 12:38:29.938313 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 12:38:29.938440 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14978,"mutex_wait_us":26,"thread_start_us":15532,"threads_started":1}
I20260811 12:38:29.944135 5208 tablet_service.cc:1467] Processing CreateTablet for tablet aee1a5d683c44b4fb09ea25187402cc0 (DEFAULT_TABLE table=test-table [id=c8b768dc167c4f37ba9ae8e3bdb59790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:29.945756 5208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee1a5d683c44b4fb09ea25187402cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:29.946288 5463 tablet_service.cc:1467] Processing CreateTablet for tablet aee1a5d683c44b4fb09ea25187402cc0 (DEFAULT_TABLE table=test-table [id=c8b768dc167c4f37ba9ae8e3bdb59790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:29.947476 5463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee1a5d683c44b4fb09ea25187402cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:29.948701 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.928784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.0.0.1:35656)
0811 12:38:29.939715 (+ 10931us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:29.939735 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:29.940226 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:29.942155 (+ 1929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:29.942160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:29.942188 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:29.942352 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.942362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.944389 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.944394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.946098 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:29.946114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.947885 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.947893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:29.948117 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.948153 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:38:29.948260 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 12:38:29.948417 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10828,"thread_start_us":13876,"threads_started":1}
I20260811 12:38:29.948877 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:29.929147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.3.204.65:46613)
0811 12:38:29.941427 (+ 12280us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:29.941436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:29.941497 (+ 61us) server_negotiation.cc:407] Connection header received
0811 12:38:29.941614 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:29.941623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:29.941913 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:29.942137 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:29.942540 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.944223 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.946314 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:29.947701 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:29.947843 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:29.948125 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:29.948318 (+ 193us) server_negotiation.cc:299] Negotiation successful
0811 12:38:29.948514 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12065,"thread_start_us":13203,"threads_started":1}
I20260811 12:38:29.952863 4973 tablet_service.cc:1467] Processing CreateTablet for tablet aee1a5d683c44b4fb09ea25187402cc0 (DEFAULT_TABLE table=test-table [id=c8b768dc167c4f37ba9ae8e3bdb59790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:29.954515 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee1a5d683c44b4fb09ea25187402cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:29.958357 5624 tablet_bootstrap.cc:492] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Bootstrap starting.
I20260811 12:38:29.961717 5625 tablet_bootstrap.cc:492] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Bootstrap starting.
I20260811 12:38:29.961790 5624 tablet_bootstrap.cc:654] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:29.963153 5624 log.cc:824] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:29.965829 5625 tablet_bootstrap.cc:654] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:29.966991 5625 log.cc:824] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:29.968941 5624 tablet_bootstrap.cc:492] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: No bootstrap required, opened a new log
I20260811 12:38:29.969292 5624 ts_tablet_manager.cc:1397] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260811 12:38:29.970161 5625 tablet_bootstrap.cc:492] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: No bootstrap required, opened a new log
I20260811 12:38:29.970439 5625 ts_tablet_manager.cc:1397] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 12:38:29.979868 5624 raft_consensus.cc:348] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:29.980317 5628 tablet_bootstrap.cc:492] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b: Bootstrap starting.
I20260811 12:38:29.980394 5624 raft_consensus.cc:374] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:29.980554 5624 raft_consensus.cc:729] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766488a9bdd1416e8b7a06659e9a3dec, State: Initialized, Role: FOLLOWER
I20260811 12:38:29.980662 5625 raft_consensus.cc:348] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:29.981088 5625 raft_consensus.cc:374] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:29.981184 5625 raft_consensus.cc:729] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d8af92656442048c4a41708a3f22df, State: Initialized, Role: FOLLOWER
I20260811 12:38:29.981284 5624 consensus_queue.cc:260] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:29.981771 5625 consensus_queue.cc:260] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:29.982676 5332 heartbeater.cc:503] Master 127.3.204.126:33257 was elected leader, sending a full tablet report...
I20260811 12:38:29.983465 5624 ts_tablet_manager.cc:1428] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260811 12:38:29.984426 5628 tablet_bootstrap.cc:654] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:29.984549 5594 heartbeater.cc:503] Master 127.3.204.126:33257 was elected leader, sending a full tablet report...
I20260811 12:38:29.985090 5625 ts_tablet_manager.cc:1428] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260811 12:38:29.985984 5628 log.cc:824] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:29.989238 5628 tablet_bootstrap.cc:492] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b: No bootstrap required, opened a new log
I20260811 12:38:29.989547 5628 ts_tablet_manager.cc:1397] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20260811 12:38:29.997006 5628 raft_consensus.cc:348] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:29.997413 5628 raft_consensus.cc:374] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:29.997524 5628 raft_consensus.cc:729] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3243b7a2c11342a7983a3b1aca49ac2b, State: Initialized, Role: FOLLOWER
I20260811 12:38:29.998188 5628 consensus_queue.cc:260] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:29.999950 5628 ts_tablet_manager.cc:1428] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260811 12:38:30.074931 5631 raft_consensus.cc:482] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:30.075320 5631 raft_consensus.cc:504] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:30.077080 5631 leader_election.cc:288] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d8af92656442048c4a41708a3f22df (127.3.204.67:44309), 3243b7a2c11342a7983a3b1aca49ac2b (127.3.204.65:46613)
I20260811 12:38:30.084558 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.077518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:33335 (local address 127.3.204.67:44309)
0811 12:38:30.077731 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:30.077739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:30.078133 (+ 394us) server_negotiation.cc:407] Connection header received
0811 12:38:30.078577 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:30.078584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:30.078659 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:30.078712 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:30.079499 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.080496 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.082596 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.083488 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.083649 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:30.083724 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:30.084128 (+ 404us) server_negotiation.cc:299] Negotiation successful
0811 12:38:30.084268 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:38:30.084566 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.077349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:44309 (local address 127.3.204.66:33335)
0811 12:38:30.078018 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.078034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:30.078320 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:30.078901 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:30.078904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:30.078929 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:30.079328 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:30.079340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.080660 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.080666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:30.082376 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:30.082391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.083708 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.083716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:30.083893 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:30.083935 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 12:38:30.084037 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 12:38:30.084186 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":22,"thread_start_us":353,"threads_started":1}
I20260811 12:38:30.084705 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.078299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:51087 (local address 127.3.204.65:46613)
0811 12:38:30.078528 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:30.078534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:30.078848 (+ 314us) server_negotiation.cc:407] Connection header received
0811 12:38:30.079006 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:30.079011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:30.079101 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:30.079190 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:30.079883 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.080811 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.082863 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.083754 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.083916 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:30.083992 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:30.084294 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 12:38:30.084419 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:38:30.084849 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.078162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.3.204.66:51087)
0811 12:38:30.078739 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.078754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:30.078886 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:30.079231 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:30.079238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:30.079262 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:30.079698 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:30.079712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.080985 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.080989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:30.082674 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:30.082690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.083917 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.083922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:30.084115 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:30.084149 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 12:38:30.084234 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 12:38:30.084353 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":34,"thread_start_us":301,"threads_started":1}
I20260811 12:38:30.085886 5003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee1a5d683c44b4fb09ea25187402cc0" candidate_uuid: "766488a9bdd1416e8b7a06659e9a3dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" is_pre_election: true
I20260811 12:38:30.086143 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee1a5d683c44b4fb09ea25187402cc0" candidate_uuid: "766488a9bdd1416e8b7a06659e9a3dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d8af92656442048c4a41708a3f22df" is_pre_election: true
I20260811 12:38:30.086570 5003 raft_consensus.cc:2454] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 766488a9bdd1416e8b7a06659e9a3dec in term 0.
I20260811 12:38:30.086696 5496 raft_consensus.cc:2454] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 766488a9bdd1416e8b7a06659e9a3dec in term 0.
I20260811 12:38:30.087342 5128 leader_election.cc:302] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3243b7a2c11342a7983a3b1aca49ac2b, 766488a9bdd1416e8b7a06659e9a3dec; no voters:
I20260811 12:38:30.087740 5631 raft_consensus.cc:2781] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:38:30.087918 5631 raft_consensus.cc:482] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:38:30.088034 5631 raft_consensus.cc:3037] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:30.090255 5631 raft_consensus.cc:504] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:30.090839 5631 leader_election.cc:288] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [CANDIDATE]: Term 1 election: Requested vote from peers a7d8af92656442048c4a41708a3f22df (127.3.204.67:44309), 3243b7a2c11342a7983a3b1aca49ac2b (127.3.204.65:46613)
I20260811 12:38:30.091511 5003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee1a5d683c44b4fb09ea25187402cc0" candidate_uuid: "766488a9bdd1416e8b7a06659e9a3dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3243b7a2c11342a7983a3b1aca49ac2b"
I20260811 12:38:30.091503 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee1a5d683c44b4fb09ea25187402cc0" candidate_uuid: "766488a9bdd1416e8b7a06659e9a3dec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d8af92656442048c4a41708a3f22df"
I20260811 12:38:30.091746 5496 raft_consensus.cc:3037] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:30.091753 5003 raft_consensus.cc:3037] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:30.095127 5496 raft_consensus.cc:2454] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 766488a9bdd1416e8b7a06659e9a3dec in term 1.
I20260811 12:38:30.095127 5003 raft_consensus.cc:2454] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 766488a9bdd1416e8b7a06659e9a3dec in term 1.
I20260811 12:38:30.095698 5128 leader_election.cc:302] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 766488a9bdd1416e8b7a06659e9a3dec, a7d8af92656442048c4a41708a3f22df; no voters:
I20260811 12:38:30.096020 5631 raft_consensus.cc:2781] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:30.096563 5631 raft_consensus.cc:686] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 1 LEADER]: Becoming Leader. State: Replica: 766488a9bdd1416e8b7a06659e9a3dec, State: Running, Role: LEADER
I20260811 12:38:30.097121 5631 consensus_queue.cc:237] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } }
I20260811 12:38:30.101979 4745 catalog_manager.cc:5495] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec reported cstate change: term changed from 0 to 1, leader changed from <none> to 766488a9bdd1416e8b7a06659e9a3dec (127.3.204.66). New cstate: current_term: 1 leader_uuid: "766488a9bdd1416e8b7a06659e9a3dec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "766488a9bdd1416e8b7a06659e9a3dec" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 36303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 } health_report { overall_health: UNKNOWN } } }
W20260811 12:38:30.106976 5595 tablet.cc:2369] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:38:30.118817 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.113175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.3.204.66:36303)
0811 12:38:30.113397 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:30.113403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:30.113513 (+ 110us) server_negotiation.cc:407] Connection header received
0811 12:38:30.113735 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:30.113739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:30.113834 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:30.113980 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:38:30.114363 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.115075 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.116510 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.116924 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.117044 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:30.117663 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:38:30.118193 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:38:30.118358 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:30.118532 (+ 174us) server_negotiation.cc:299] Negotiation successful
0811 12:38:30.118578 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260811 12:38:30.120488 5333 tablet.cc:2369] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:38:30.127862 5003 raft_consensus.cc:1261] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 1 FOLLOWER]: Refusing update from remote peer 766488a9bdd1416e8b7a06659e9a3dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:30.127854 5496 raft_consensus.cc:1261] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 1 FOLLOWER]: Refusing update from remote peer 766488a9bdd1416e8b7a06659e9a3dec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:30.129031 5639 consensus_queue.cc:1035] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d8af92656442048c4a41708a3f22df" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 44309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:30.129406 5631 consensus_queue.cc:1035] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "3243b7a2c11342a7983a3b1aca49ac2b" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 46613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:30.143666 5648 mvcc.cc:205] Tried to move back new op lower bound from 7317307023872856064 to 7317307023762227200. Current Snapshot: MvccSnapshot[applied={T|T < 7317307023872856064}]
I20260811 12:38:30.157020 4743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:30.157308 4743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:30.161510 4743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4635655b82c44aab84ec7683de251c93: Sending DeleteTablet for 3 replicas of tablet aee1a5d683c44b4fb09ea25187402cc0
I20260811 12:38:30.162844 5208 tablet_service.cc:1514] Processing DeleteTablet for tablet aee1a5d683c44b4fb09ea25187402cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:30 UTC) from {username='slave'} at 127.0.0.1:56222
I20260811 12:38:30.162856 5463 tablet_service.cc:1514] Processing DeleteTablet for tablet aee1a5d683c44b4fb09ea25187402cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:30 UTC) from {username='slave'} at 127.0.0.1:57326
I20260811 12:38:30.163710 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet aee1a5d683c44b4fb09ea25187402cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:30 UTC) from {username='slave'} at 127.0.0.1:35656
I20260811 12:38:30.164925 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 4853
I20260811 12:38:30.211885 5669 tablet_replica.cc:330] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: stopping tablet replica
I20260811 12:38:30.212598 5669 raft_consensus.cc:2229] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:38:30.220722 5669 raft_consensus.cc:2258] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:30.221459 5670 tablet_replica.cc:330] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b: stopping tablet replica
I20260811 12:38:30.221944 5670 raft_consensus.cc:2229] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:30.222473 5670 raft_consensus.cc:2258] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:30.223934 5666 tablet_replica.cc:330] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: stopping tablet replica
I20260811 12:38:30.224381 5666 raft_consensus.cc:2229] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:30.224913 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.166416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.3.204.65:46613)
0811 12:38:30.166615 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:30.166621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:30.206414 (+ 39793us) server_negotiation.cc:407] Connection header received
0811 12:38:30.206999 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:30.207006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:30.207142 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:30.207227 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:30.207972 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.209279 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.212924 (+ 3645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.213625 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.213732 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:30.223241 (+ 9509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:30.223253 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:30.223271 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:30.224006 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:30.224102 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:30.224133 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:30.224249 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:30.224438 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 12:38:30.224566 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 12:38:30.226176 5081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:30.226590 5669 ts_tablet_manager.cc:1905] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:30.234463 5669 ts_tablet_manager.cc:1918] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:30.234697 5669 log.cc:1196] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/aee1a5d683c44b4fb09ea25187402cc0
I20260811 12:38:30.235321 5669 ts_tablet_manager.cc:1939] T aee1a5d683c44b4fb09ea25187402cc0 P 766488a9bdd1416e8b7a06659e9a3dec: Deleting consensus metadata
I20260811 12:38:30.235301 5666 raft_consensus.cc:2258] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:30.237470 4710 catalog_manager.cc:4841] TS 766488a9bdd1416e8b7a06659e9a3dec (127.3.204.66:36303): tablet aee1a5d683c44b4fb09ea25187402cc0 (table test-table [id=c8b768dc167c4f37ba9ae8e3bdb59790]) successfully deleted
I20260811 12:38:30.237951 5666 ts_tablet_manager.cc:1905] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:30.245366 5666 ts_tablet_manager.cc:1918] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:30.245572 5666 log.cc:1196] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/aee1a5d683c44b4fb09ea25187402cc0
I20260811 12:38:30.246181 5666 ts_tablet_manager.cc:1939] T aee1a5d683c44b4fb09ea25187402cc0 P a7d8af92656442048c4a41708a3f22df: Deleting consensus metadata
I20260811 12:38:30.248291 4715 catalog_manager.cc:4841] TS a7d8af92656442048c4a41708a3f22df (127.3.204.67:44309): tablet aee1a5d683c44b4fb09ea25187402cc0 (table test-table [id=c8b768dc167c4f37ba9ae8e3bdb59790]) successfully deleted
I20260811 12:38:30.226238 5670 ts_tablet_manager.cc:1905] T aee1a5d683c44b4fb09ea25187402cc0 P 3243b7a2c11342a7983a3b1aca49ac2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:30.320268 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 4853
W20260811 12:38:30.348896 4715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.65:46613 (error 108)
I20260811 12:38:30.349174 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 5094
I20260811 12:38:30.356256 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.355705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.0.0.1:35678)
0811 12:38:30.355888 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.356024 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 12:38:30.356578 4715 catalog_manager.cc:4584] TS 3243b7a2c11342a7983a3b1aca49ac2b (127.3.204.65:46613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aee1a5d683c44b4fb09ea25187402cc0: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46613: connect: Connection refused (error 111)
I20260811 12:38:30.358719 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.350747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.3.204.66:36303)
0811 12:38:30.350952 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:30.350958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:30.350978 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:38:30.351044 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:30.351049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:30.351163 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:30.351273 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:30.351962 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.353188 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.355159 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.355795 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.355934 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:30.357065 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:30.357077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:30.357100 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:30.357905 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:30.357967 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:30.358000 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:30.358109 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:30.358232 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 12:38:30.358340 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 12:38:30.359422 5321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:30.410383 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.409898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.0.0.1:35682)
0811 12:38:30.410142 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.410234 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260811 12:38:30.411432 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.411047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.0.0.1:35692)
0811 12:38:30.411204 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.411276 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 12:38:30.472124 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.471611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.0.0.1:35696)
0811 12:38:30.471803 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.471919 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 12:38:30.472327 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 5094
I20260811 12:38:30.477414 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.476972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.0.0.1:35708)
0811 12:38:30.477155 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.477238 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 12:38:30.497665 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 5337
I20260811 12:38:30.504889 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.498512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.3.204.67:44309)
0811 12:38:30.498699 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:30.498706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:30.498775 (+ 69us) server_negotiation.cc:407] Connection header received
0811 12:38:30.498977 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:30.498981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:30.499080 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:30.499170 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:30.499634 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.500473 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.501809 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:30.502210 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:30.502322 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:30.503396 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:30.503408 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:30.503423 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:30.503995 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:30.504045 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:30.504068 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:30.504316 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:30.504438 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 12:38:30.504557 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:38:30.505548 5586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:30.558260 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.557669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.0.0.1:35722)
0811 12:38:30.557943 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.558079 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260811 12:38:30.559459 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:30.559011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:46613 (local address 127.0.0.1:35732)
0811 12:38:30.559205 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:30.559289 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260811 12:38:30.594521 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 5337
I20260811 12:38:30.617733 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 4679
I20260811 12:38:30.618866 4795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:30.727685 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 4679
2026-08-11T12:38:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2929 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:30 dist-test-slave-lgr5 krb5kdc[5682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:30 dist-test-slave-lgr5 krb5kdc[5682](info): set up 2 sockets
Aug 11 12:38:30 dist-test-slave-lgr5 krb5kdc[5682](info): commencing operation
krb5kdc: starting...
W20260811 12:38:32.031416 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.001s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:32 dist-test-slave-lgr5 krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:38:32.347791 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:40887
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:40887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:32.698052 5705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:32.698359 5705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:32.698446 5705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:32.705724 5705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:32.705842 5705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:32.705888 5705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:32.705943 5705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:32.705994 5705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:32.715998 5705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:40887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:40887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:32.717353 5705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:32.718690 5705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:32.730613 5716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:32.730680 5715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:32.731186 5705 server_base.cc:1034] running on GCE node
W20260811 12:38:32.731377 5720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:32.732273 5705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:32.733740 5705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:32.735008 5705 hybrid_clock.cc:648] HybridClock initialized: now 1786451912734931 us; error 60 us; skew 500 ppm
I20260811 12:38:32.735376 5705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:32 dist-test-slave-lgr5 krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451912, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:32.742086 5705 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:38:32.743794 5705 webserver.cc:460] Webserver started at http://127.3.204.126:37237/ using document root <none> and password file <none>
I20260811 12:38:32.744637 5705 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:32.744792 5705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:32.745137 5705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:32.748572 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "2984d83ad13349ad94ee4892deca0342"
format_stamp: "Formatted at 2026-08-11 12:38:32 on dist-test-slave-lgr5"
server_key: "a74179bb78702b8ff81cc34be4af2aaa"
server_key_iv: "d0d058d71c3302b722b2e4ab8909a420"
server_key_version: "encryptionkey@0"
I20260811 12:38:32.749432 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "2984d83ad13349ad94ee4892deca0342"
format_stamp: "Formatted at 2026-08-11 12:38:32 on dist-test-slave-lgr5"
server_key: "a74179bb78702b8ff81cc34be4af2aaa"
server_key_iv: "d0d058d71c3302b722b2e4ab8909a420"
server_key_version: "encryptionkey@0"
I20260811 12:38:32.755478 5705 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260811 12:38:32.759509 5728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:32.760970 5705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:38:32.761163 5705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "2984d83ad13349ad94ee4892deca0342"
format_stamp: "Formatted at 2026-08-11 12:38:32 on dist-test-slave-lgr5"
server_key: "a74179bb78702b8ff81cc34be4af2aaa"
server_key_iv: "d0d058d71c3302b722b2e4ab8909a420"
server_key_version: "encryptionkey@0"
I20260811 12:38:32.761363 5705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:32.788203 5705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:32.793605 5705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:32.793919 5705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:32.812042 5705 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:40887
I20260811 12:38:32.812062 5829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:40887 every 8 connection(s)
I20260811 12:38:32.813539 5705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:32.817452 5830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:32.819717 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 5705
I20260811 12:38:32.819907 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:32.820482 3889 external_mini_cluster.cc:1428] Setting key 8d6b5391525a01a5d236e961ce850080
I20260811 12:38:32.829209 5830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342: Bootstrap starting.
I20260811 12:38:32.832275 5830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:32.833253 5830 log.cc:824] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342: Log is configured to *not* fsync() on all Append() calls
Aug 11 12:38:32 dist-test-slave-lgr5 krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:32.836344 5830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342: No bootstrap required, opened a new log
I20260811 12:38:32.842990 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:32.822567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.3.204.126:40887)
0811 12:38:32.823394 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:32.823407 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:32.823450 (+ 43us) server_negotiation.cc:407] Connection header received
0811 12:38:32.825213 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:32.825237 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:32.825616 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:32.825972 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:32.827479 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:32.828853 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:32.830137 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:32.830855 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:32.831033 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:32.835489 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:32.835497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:32.835512 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:32.835551 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:32.838639 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:32.839528 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:32.839535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:32.839542 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:32.839756 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:32.840225 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:32.840229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:32.840231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:32.840803 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:32.841037 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:32.841509 (+ 472us) server_negotiation.cc:299] Negotiation successful
0811 12:38:32.841916 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":367,"threads_started":1}
I20260811 12:38:32.844967 5830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984d83ad13349ad94ee4892deca0342" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40887 } }
I20260811 12:38:32.845328 5830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:32.845428 5830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2984d83ad13349ad94ee4892deca0342, State: Initialized, Role: FOLLOWER
I20260811 12:38:32.845973 5830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984d83ad13349ad94ee4892deca0342" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40887 } }
I20260811 12:38:32.846202 5830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:32.846325 5830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:32.846477 5830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:32.848690 5830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984d83ad13349ad94ee4892deca0342" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40887 } }
I20260811 12:38:32.849192 5830 leader_election.cc:302] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2984d83ad13349ad94ee4892deca0342; no voters:
I20260811 12:38:32.849763 5830 leader_election.cc:288] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:32.849925 5836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:32.850605 5836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [term 1 LEADER]: Becoming Leader. State: Replica: 2984d83ad13349ad94ee4892deca0342, State: Running, Role: LEADER
I20260811 12:38:32.851156 5836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984d83ad13349ad94ee4892deca0342" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40887 } }
I20260811 12:38:32.851903 5830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:32.854110 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2984d83ad13349ad94ee4892deca0342. Latest consensus state: current_term: 1 leader_uuid: "2984d83ad13349ad94ee4892deca0342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984d83ad13349ad94ee4892deca0342" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40887 } } }
I20260811 12:38:32.854477 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:32.855049 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2984d83ad13349ad94ee4892deca0342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984d83ad13349ad94ee4892deca0342" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40887 } } }
I20260811 12:38:32.855412 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:32.855423 5845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:32.869743 5845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:32.882174 5845 catalog_manager.cc:1347] Generated new cluster ID: 348dc181a98b497aa4b7a01837ed4db8
I20260811 12:38:32.882301 5845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:32.899327 5845 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:32.900767 5845 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:32.913264 5845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342: Generated new TSK 0
I20260811 12:38:32.914156 5845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
Entry for principal kudu/127.3.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab.
I20260811 12:38:32.984643 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40887
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:33.332584 5874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:33.332893 5874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:33.332988 5874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:33.340420 5874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:33.340543 5874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:33.340656 5874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:38:33.350602 5874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.65.keytab
--principal=kudu/127.3.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:33.351960 5874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:33.353370 5874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:33.366974 5884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:33.367057 5890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:33.367057 5885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:33.367846 5874 server_base.cc:1034] running on GCE node
I20260811 12:38:33.368496 5874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:33.369441 5874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:33.370659 5874 hybrid_clock.cc:648] HybridClock initialized: now 1786451913370605 us; error 43 us; skew 500 ppm
I20260811 12:38:33.370980 5874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:33 dist-test-slave-lgr5 krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:33.374245 5874 init.cc:377] Logged in from keytab as kudu/127.3.204.65@KRBTEST.COM (short username kudu)
I20260811 12:38:33.375849 5874 webserver.cc:460] Webserver started at http://127.3.204.65:45883/ using document root <none> and password file <none>
I20260811 12:38:33.376619 5874 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:33.376739 5874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:33.377049 5874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:33.380019 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "919722c1774c4f8dad480c70659fb548"
format_stamp: "Formatted at 2026-08-11 12:38:33 on dist-test-slave-lgr5"
server_key: "196df6f679cc1609733522449bacd1f8"
server_key_iv: "6b21c555083e02af47e0f01b2b32c660"
server_key_version: "encryptionkey@0"
I20260811 12:38:33.380858 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "919722c1774c4f8dad480c70659fb548"
format_stamp: "Formatted at 2026-08-11 12:38:33 on dist-test-slave-lgr5"
server_key: "196df6f679cc1609733522449bacd1f8"
server_key_iv: "6b21c555083e02af47e0f01b2b32c660"
server_key_version: "encryptionkey@0"
I20260811 12:38:33.386288 5874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260811 12:38:33.389910 5903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:33.391295 5874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:38:33.391484 5874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "919722c1774c4f8dad480c70659fb548"
format_stamp: "Formatted at 2026-08-11 12:38:33 on dist-test-slave-lgr5"
server_key: "196df6f679cc1609733522449bacd1f8"
server_key_iv: "6b21c555083e02af47e0f01b2b32c660"
server_key_version: "encryptionkey@0"
I20260811 12:38:33.391695 5874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:33.409112 5874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:33.413484 5874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:33.413762 5874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:33.414716 5874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:33.416237 5874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:33.416317 5874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:33.416443 5874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:33.416524 5874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:33.447772 5874 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:43909
I20260811 12:38:33.447994 6100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:43909 every 8 connection(s)
I20260811 12:38:33.449141 5874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:38:33.456780 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 5874
I20260811 12:38:33.456964 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:38:33.457403 3889 external_mini_cluster.cc:1428] Setting key 3347dcdc53e63c23591f086eb186fbd2
Aug 11 12:38:33 dist-test-slave-lgr5 krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.65@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:33.472152 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:33.453127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:48777 (local address 127.3.204.126:40887)
0811 12:38:33.453807 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:33.453814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:33.454396 (+ 582us) server_negotiation.cc:407] Connection header received
0811 12:38:33.456352 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:33.456358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:33.456456 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:33.456597 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:33.458498 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:33.459342 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:33.461093 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:33.461721 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:33.461868 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:33.465826 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:33.465835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:33.465841 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:33.465880 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:33.468479 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:33.469429 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:33.469436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:33.469439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:33.469597 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:33.470166 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:33.470171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:33.470173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:33.470483 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:33.470607 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:33.471550 (+ 943us) server_negotiation.cc:299] Negotiation successful
0811 12:38:33.471809 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":305,"threads_started":1}
I20260811 12:38:33.473424 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:33.453402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40887 (local address 127.3.204.65:48777)
0811 12:38:33.454215 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:33.454259 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:33.456022 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:33.456872 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:33.456885 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:33.457534 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:33.458298 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:33.458323 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:33.459538 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:33.459543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:33.460931 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:33.460946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:33.461981 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:33.461987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:33.462214 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:33.462929 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:33.462958 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:33.465556 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:33.468637 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:33.468647 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:33.468665 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:33.469252 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:33.469717 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:33.469721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:33.469724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:33.469986 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:33.470614 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:33.470622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:33.471079 (+ 457us) client_negotiation.cc:769] Sending connection context
0811 12:38:33.471398 (+ 319us) client_negotiation.cc:240] Negotiation successful
0811 12:38:33.471963 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":374,"threads_started":1}
I20260811 12:38:33.475018 6101 heartbeater.cc:347] Connected to a master server at 127.3.204.126:40887
I20260811 12:38:33.475359 6101 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:33.476289 6101 heartbeater.cc:511] Master 127.3.204.126:40887 requested a full tablet report, sending...
I20260811 12:38:33.478613 5758 ts_manager.cc:194] Registered new tserver with Master: 919722c1774c4f8dad480c70659fb548 (127.3.204.65:43909)
I20260811 12:38:33.480147 5758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.65@KRBTEST.COM'} at 127.3.204.65:48777
WARNING: no policy specified for kudu/127.3.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
Entry for principal kudu/127.3.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab.
I20260811 12:38:33.529700 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40887
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:33.872222 6110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:33.872545 6110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:33.872704 6110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:33.880303 6110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:33.880414 6110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:33.880559 6110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:38:33.891193 6110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.66.keytab
--principal=kudu/127.3.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:33.892565 6110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:33.893849 6110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:33.906977 6119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:33.906881 6122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:33.906945 6127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:33.907773 6110 server_base.cc:1034] running on GCE node
I20260811 12:38:33.908438 6110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:33.909397 6110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:33.910647 6110 hybrid_clock.cc:648] HybridClock initialized: now 1786451913910588 us; error 43 us; skew 500 ppm
I20260811 12:38:33.910993 6110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:33 dist-test-slave-lgr5 krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:33.914245 6110 init.cc:377] Logged in from keytab as kudu/127.3.204.66@KRBTEST.COM (short username kudu)
I20260811 12:38:33.915761 6110 webserver.cc:460] Webserver started at http://127.3.204.66:44407/ using document root <none> and password file <none>
I20260811 12:38:33.916508 6110 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:33.916630 6110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:33.916944 6110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:33.919824 6110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "beb807c78ecb4eb689307f2929f61db3"
format_stamp: "Formatted at 2026-08-11 12:38:33 on dist-test-slave-lgr5"
server_key: "8b368a33613394a3f2d63cc7361278cb"
server_key_iv: "2b0c3dd2956fed99eb879c49c8772b20"
server_key_version: "encryptionkey@0"
I20260811 12:38:33.920672 6110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "beb807c78ecb4eb689307f2929f61db3"
format_stamp: "Formatted at 2026-08-11 12:38:33 on dist-test-slave-lgr5"
server_key: "8b368a33613394a3f2d63cc7361278cb"
server_key_iv: "2b0c3dd2956fed99eb879c49c8772b20"
server_key_version: "encryptionkey@0"
I20260811 12:38:33.925722 6110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260811 12:38:33.929271 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:33.930655 6110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:38:33.930830 6110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "beb807c78ecb4eb689307f2929f61db3"
format_stamp: "Formatted at 2026-08-11 12:38:33 on dist-test-slave-lgr5"
server_key: "8b368a33613394a3f2d63cc7361278cb"
server_key_iv: "2b0c3dd2956fed99eb879c49c8772b20"
server_key_version: "encryptionkey@0"
I20260811 12:38:33.931036 6110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:33.954898 6110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:33.959188 6110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:33.959496 6110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:33.960634 6110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:33.962074 6110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:33.962159 6110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:33.962278 6110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:33.962330 6110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:33.996906 6110 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:38995
I20260811 12:38:33.996968 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:38995 every 8 connection(s)
I20260811 12:38:33.998275 6110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:38:34.001116 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 6110
I20260811 12:38:34.001317 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:38:34.001971 3889 external_mini_cluster.cc:1428] Setting key a11ca0194b19be89d8fc16ed1c3852e1
Aug 11 12:38:34 dist-test-slave-lgr5 krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.66@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:34.020829 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.001121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:32959 (local address 127.3.204.126:40887)
0811 12:38:34.003533 (+ 2412us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.003537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.003555 (+ 18us) server_negotiation.cc:407] Connection header received
0811 12:38:34.005075 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.005082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.005169 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.005292 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:34.006897 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.007886 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.009596 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.010158 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.010310 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.014289 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:34.014296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:34.014301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:34.014345 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:34.017432 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:34.018345 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:34.018349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:34.018351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:34.018496 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:34.018999 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:34.019003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:34.019005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:34.019295 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:34.019407 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.020338 (+ 931us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.020524 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2314,"thread_start_us":2214,"threads_started":1}
I20260811 12:38:34.022269 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.001652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40887 (local address 127.3.204.66:32959)
0811 12:38:34.002584 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:34.002636 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:34.004763 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:34.005514 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:34.005522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:34.006093 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:34.006706 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.006726 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.008066 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.008070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.009432 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.009447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.010405 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.010411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.010631 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.011397 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:34.011423 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:34.013994 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:34.017605 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:34.017614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:34.017630 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:34.018203 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:34.018605 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:34.018609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:34.018612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:34.018877 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:34.019426 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:34.019434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:34.019891 (+ 457us) client_negotiation.cc:769] Sending connection context
0811 12:38:34.020234 (+ 343us) client_negotiation.cc:240] Negotiation successful
0811 12:38:34.020815 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":416,"threads_started":1}
I20260811 12:38:34.023967 6342 heartbeater.cc:347] Connected to a master server at 127.3.204.126:40887
I20260811 12:38:34.024312 6342 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:34.025065 6342 heartbeater.cc:511] Master 127.3.204.126:40887 requested a full tablet report, sending...
I20260811 12:38:34.026646 5758 ts_manager.cc:194] Registered new tserver with Master: beb807c78ecb4eb689307f2929f61db3 (127.3.204.66:38995)
I20260811 12:38:34.027704 5758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.66@KRBTEST.COM'} at 127.3.204.66:32959
WARNING: no policy specified for kudu/127.3.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
Entry for principal kudu/127.3.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab.
I20260811 12:38:34.075230 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40887
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:34.422945 6354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:34.423246 6354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:34.423381 6354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:34.430828 6354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:34.430956 6354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:34.431118 6354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:38:34.441152 6354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.67.keytab
--principal=kudu/127.3.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:40887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:34.442467 6354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:34.443789 6354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:34.457582 6363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:34.457525 6366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:34.457520 6364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:34.458380 6354 server_base.cc:1034] running on GCE node
I20260811 12:38:34.458989 6354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:34.459949 6354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:34.461161 6354 hybrid_clock.cc:648] HybridClock initialized: now 1786451914461109 us; error 63 us; skew 500 ppm
I20260811 12:38:34.461488 6354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:34 dist-test-slave-lgr5 krb5kdc[5682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451914, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:34.464695 6354 init.cc:377] Logged in from keytab as kudu/127.3.204.67@KRBTEST.COM (short username kudu)
I20260811 12:38:34.466215 6354 webserver.cc:460] Webserver started at http://127.3.204.67:33295/ using document root <none> and password file <none>
I20260811 12:38:34.466935 6354 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:34.467060 6354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:34.467388 6354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:34.470352 6354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6"
format_stamp: "Formatted at 2026-08-11 12:38:34 on dist-test-slave-lgr5"
server_key: "e48be274c0bfe565683b57636cc44f4c"
server_key_iv: "c3fc060e6f536e699773690c30c0097d"
server_key_version: "encryptionkey@0"
I20260811 12:38:34.471185 6354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6"
format_stamp: "Formatted at 2026-08-11 12:38:34 on dist-test-slave-lgr5"
server_key: "e48be274c0bfe565683b57636cc44f4c"
server_key_iv: "c3fc060e6f536e699773690c30c0097d"
server_key_version: "encryptionkey@0"
I20260811 12:38:34.476532 6354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260811 12:38:34.479975 6377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:34.481384 6354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:34.481559 6354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6"
format_stamp: "Formatted at 2026-08-11 12:38:34 on dist-test-slave-lgr5"
server_key: "e48be274c0bfe565683b57636cc44f4c"
server_key_iv: "c3fc060e6f536e699773690c30c0097d"
server_key_version: "encryptionkey@0"
I20260811 12:38:34.481752 6354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:34.483893 6101 heartbeater.cc:503] Master 127.3.204.126:40887 was elected leader, sending a full tablet report...
I20260811 12:38:34.497468 6354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:34.501863 6354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:34.502118 6354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:34.503119 6354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:34.504487 6354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:34.504563 6354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:34.504693 6354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:34.504745 6354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:34.536616 6354 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:43885
I20260811 12:38:34.536691 6573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:43885 every 8 connection(s)
I20260811 12:38:34.537928 6354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:38:34.547928 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 6354
I20260811 12:38:34.548161 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:38:34.548626 3889 external_mini_cluster.cc:1428] Setting key cea1c85eea95cf4f42117d4946ee6566
Aug 11 12:38:34 dist-test-slave-lgr5 krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451914, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.67@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:38:34.558080 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.540509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:48495 (local address 127.3.204.126:40887)
0811 12:38:34.540942 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.540948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.541938 (+ 990us) server_negotiation.cc:407] Connection header received
0811 12:38:34.543829 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.543835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.543929 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.544030 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:34.545586 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.546394 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.548192 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.548638 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.548758 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.552696 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:34.552704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:34.552711 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:34.552755 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:34.555303 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:34.555976 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:34.555979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:34.555982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:34.556129 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:34.556517 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:34.556520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:34.556522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:34.556778 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:34.556907 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.557591 (+ 684us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.557764 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":224,"threads_started":1}
I20260811 12:38:34.559101 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.541026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40887 (local address 127.3.204.67:48495)
0811 12:38:34.541768 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:34.541807 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:34.543546 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:34.544310 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:34.544319 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:34.544844 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:34.545409 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.545427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.546571 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.546575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.547999 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.548016 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.548891 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.548900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.549135 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.549887 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:38:34.549910 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:34.552478 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:34.555435 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:34.555442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:34.555454 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:34.555870 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:34.556219 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:38:34.556223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:38:34.556225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:38:34.556433 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:38:34.556916 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:34.556923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:34.557253 (+ 330us) client_negotiation.cc:769] Sending connection context
0811 12:38:34.557518 (+ 265us) client_negotiation.cc:240] Negotiation successful
0811 12:38:34.557922 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":286,"threads_started":1}
I20260811 12:38:34.560503 6575 heartbeater.cc:347] Connected to a master server at 127.3.204.126:40887
I20260811 12:38:34.560797 6575 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:34.561544 6575 heartbeater.cc:511] Master 127.3.204.126:40887 requested a full tablet report, sending...
I20260811 12:38:34.563086 5758 ts_manager.cc:194] Registered new tserver with Master: 20fadd2aa46c4ad8bcd401b807ae7ba6 (127.3.204.67:43885)
I20260811 12:38:34.564042 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:38:34.564184 5758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.204.67@KRBTEST.COM'} at 127.3.204.67:48495
I20260811 12:38:34.581377 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.569023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.3.204.126:40887)
0811 12:38:34.569183 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.569188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.569607 (+ 419us) server_negotiation.cc:407] Connection header received
0811 12:38:34.569747 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.569750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.569814 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.569907 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:34.571280 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.572191 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.573536 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.573940 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.574028 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.576309 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:34.576314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:34.576318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:34.576350 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:34.578841 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:34.579480 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:34.579485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:34.579487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:34.579612 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:34.579916 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:34.579919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:34.579922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:34.580294 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:34.580478 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.580846 (+ 368us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.581052 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:38:34.585192 5758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:38:34.592183 5758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:38:34.615170 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.603931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38314)
0811 12:38:34.605945 (+ 2014us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:34.605964 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:34.606079 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:34.607618 (+ 1539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:34.607627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:34.607662 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:34.608106 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.608121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.610017 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.610022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.612304 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.612320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.614163 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.614170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.614375 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.614439 (+ 64us) client_negotiation.cc:769] Sending connection context
0811 12:38:34.614546 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 12:38:34.614730 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1885,"mutex_wait_us":23,"thread_start_us":1747,"threads_started":1}
I20260811 12:38:34.615986 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.604535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.3.204.65:43909)
0811 12:38:34.606652 (+ 2117us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.606663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.606695 (+ 32us) server_negotiation.cc:407] Connection header received
0811 12:38:34.606802 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.606811 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.607074 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.607278 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:34.608312 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.609879 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.612516 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.613939 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.614076 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.615368 (+ 1292us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.615471 (+ 103us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.615652 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1933,"thread_start_us":1756,"threads_started":1}
I20260811 12:38:34.616127 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.603872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:38995 (local address 127.0.0.1:40630)
0811 12:38:34.604577 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:34.604613 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:34.604779 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:34.609376 (+ 4597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:34.609381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:34.609403 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:34.609902 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.609917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.611324 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.611329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.613234 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.613250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.615322 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.615329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.615494 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.615525 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 12:38:34.615619 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 12:38:34.615752 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":318,"threads_started":1}
I20260811 12:38:34.617102 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.604073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.3.204.66:38995)
0811 12:38:34.608557 (+ 4484us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.608566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.608593 (+ 27us) server_negotiation.cc:407] Connection header received
0811 12:38:34.608689 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.608695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.608928 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.609124 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:34.610099 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.611192 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.613406 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.614796 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.614939 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.615176 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.616578 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.616755 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4311,"thread_start_us":4178,"threads_started":1}
I20260811 12:38:34.617112 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.605764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43885 (local address 127.0.0.1:51594)
0811 12:38:34.608863 (+ 3099us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:34.608881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:34.608996 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:34.609649 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:34.609653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:34.609686 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:34.610198 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.610214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.612344 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.612350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.614077 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.614095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.616180 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.616186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.616488 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.616514 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 12:38:34.616599 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 12:38:34.616721 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2984,"mutex_wait_us":1,"thread_start_us":2827,"threads_started":1}
I20260811 12:38:34.618044 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.606385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.3.204.67:43885)
0811 12:38:34.607185 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.607195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.609128 (+ 1933us) server_negotiation.cc:407] Connection header received
0811 12:38:34.609230 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.609237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.609447 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.609658 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:34.610426 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.612004 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.614279 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.615668 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.615789 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.616013 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.617619 (+ 1606us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.617766 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":435,"threads_started":1}
I20260811 12:38:34.619889 5994 tablet_service.cc:1467] Processing CreateTablet for tablet c3fafa0a7ec541e9a42c826e4d0a821d (DEFAULT_TABLE table=test-table [id=2793f1ede00f48289b63634fbb2e5422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:34.621512 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3fafa0a7ec541e9a42c826e4d0a821d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:34.621135 6222 tablet_service.cc:1467] Processing CreateTablet for tablet c3fafa0a7ec541e9a42c826e4d0a821d (DEFAULT_TABLE table=test-table [id=2793f1ede00f48289b63634fbb2e5422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:34.621644 6451 tablet_service.cc:1467] Processing CreateTablet for tablet c3fafa0a7ec541e9a42c826e4d0a821d (DEFAULT_TABLE table=test-table [id=2793f1ede00f48289b63634fbb2e5422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:34.622416 6222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3fafa0a7ec541e9a42c826e4d0a821d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:34.622993 6451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3fafa0a7ec541e9a42c826e4d0a821d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:34.633524 6621 tablet_bootstrap.cc:492] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Bootstrap starting.
I20260811 12:38:34.637473 6621 tablet_bootstrap.cc:654] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:34.638542 6621 log.cc:824] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:34.639568 6624 tablet_bootstrap.cc:492] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Bootstrap starting.
I20260811 12:38:34.641424 6622 tablet_bootstrap.cc:492] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Bootstrap starting.
I20260811 12:38:34.643293 6621 tablet_bootstrap.cc:492] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: No bootstrap required, opened a new log
I20260811 12:38:34.643540 6624 tablet_bootstrap.cc:654] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:34.643651 6621 ts_tablet_manager.cc:1397] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260811 12:38:34.644897 6624 log.cc:824] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:34.645080 6622 tablet_bootstrap.cc:654] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:34.646232 6622 log.cc:824] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:34.648177 6624 tablet_bootstrap.cc:492] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: No bootstrap required, opened a new log
I20260811 12:38:34.648478 6624 ts_tablet_manager.cc:1397] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 12:38:34.649364 6622 tablet_bootstrap.cc:492] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: No bootstrap required, opened a new log
I20260811 12:38:34.649641 6622 ts_tablet_manager.cc:1397] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 12:38:34.652191 6621 raft_consensus.cc:348] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.652556 6621 raft_consensus.cc:374] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:34.652683 6621 raft_consensus.cc:729] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beb807c78ecb4eb689307f2929f61db3, State: Initialized, Role: FOLLOWER
I20260811 12:38:34.653271 6621 consensus_queue.cc:260] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.654320 6342 heartbeater.cc:503] Master 127.3.204.126:40887 was elected leader, sending a full tablet report...
I20260811 12:38:34.658823 6621 ts_tablet_manager.cc:1428] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20260811 12:38:34.659466 6624 raft_consensus.cc:348] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.659960 6624 raft_consensus.cc:374] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:34.660106 6624 raft_consensus.cc:729] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20fadd2aa46c4ad8bcd401b807ae7ba6, State: Initialized, Role: FOLLOWER
I20260811 12:38:34.659951 6622 raft_consensus.cc:348] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.660552 6622 raft_consensus.cc:374] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:34.660669 6622 raft_consensus.cc:729] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919722c1774c4f8dad480c70659fb548, State: Initialized, Role: FOLLOWER
I20260811 12:38:34.660771 6624 consensus_queue.cc:260] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.661263 6622 consensus_queue.cc:260] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.661892 6575 heartbeater.cc:503] Master 127.3.204.126:40887 was elected leader, sending a full tablet report...
I20260811 12:38:34.662938 6622 ts_tablet_manager.cc:1428] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260811 12:38:34.663659 6624 ts_tablet_manager.cc:1428] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260811 12:38:34.702443 6103 tablet.cc:2369] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 12:38:34.750448 6343 tablet.cc:2369] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:38:34.759368 6634 raft_consensus.cc:482] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:34.759701 6634 raft_consensus.cc:504] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.761543 6634 leader_election.cc:288] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beb807c78ecb4eb689307f2929f61db3 (127.3.204.66:38995), 919722c1774c4f8dad480c70659fb548 (127.3.204.65:43909)
I20260811 12:38:34.767901 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.762102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:49101 (local address 127.3.204.65:43909)
0811 12:38:34.762324 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.762331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.762431 (+ 100us) server_negotiation.cc:407] Connection header received
0811 12:38:34.762634 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.762638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.762698 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.762771 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:34.763210 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.764302 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.766078 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.766943 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.767106 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.767194 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.767489 (+ 295us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.767623 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 12:38:34.767954 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.761890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.3.204.67:49101)
0811 12:38:34.762341 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:34.762360 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:34.762492 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:34.762823 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:34.762828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:34.762861 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:34.763047 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.763055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.764441 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.764446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.765857 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.765871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.767098 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.767104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.767316 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.767343 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 12:38:34.767460 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 12:38:34.767603 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":1}
I20260811 12:38:34.768399 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.762120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:58463 (local address 127.3.204.66:38995)
0811 12:38:34.762404 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.762412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.763104 (+ 692us) server_negotiation.cc:407] Connection header received
0811 12:38:34.763255 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.763259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.763325 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.763389 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:34.764064 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.764809 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.766655 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.767491 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.767641 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.767711 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.768003 (+ 292us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.768153 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 12:38:34.768431 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.762102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:38995 (local address 127.3.204.67:58463)
0811 12:38:34.762996 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:34.763013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:34.763136 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:34.763441 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:34.763446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:34.763475 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:34.763859 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.763873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.765007 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.765011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.766491 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:34.766524 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.767697 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.767703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:34.767859 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.767882 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 12:38:34.767968 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 12:38:34.768108 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":506,"threads_started":1}
I20260811 12:38:34.769204 6026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3fafa0a7ec541e9a42c826e4d0a821d" candidate_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "919722c1774c4f8dad480c70659fb548" is_pre_election: true
I20260811 12:38:34.769637 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3fafa0a7ec541e9a42c826e4d0a821d" candidate_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb807c78ecb4eb689307f2929f61db3" is_pre_election: true
I20260811 12:38:34.769912 6026 raft_consensus.cc:2454] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20fadd2aa46c4ad8bcd401b807ae7ba6 in term 0.
I20260811 12:38:34.770229 6260 raft_consensus.cc:2454] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20fadd2aa46c4ad8bcd401b807ae7ba6 in term 0.
I20260811 12:38:34.770751 6385 leader_election.cc:302] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20fadd2aa46c4ad8bcd401b807ae7ba6, 919722c1774c4f8dad480c70659fb548; no voters:
I20260811 12:38:34.771174 6634 raft_consensus.cc:2781] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:38:34.771358 6634 raft_consensus.cc:482] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:38:34.771477 6634 raft_consensus.cc:3037] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:34.774135 6634 raft_consensus.cc:504] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.774772 6634 leader_election.cc:288] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [CANDIDATE]: Term 1 election: Requested vote from peers beb807c78ecb4eb689307f2929f61db3 (127.3.204.66:38995), 919722c1774c4f8dad480c70659fb548 (127.3.204.65:43909)
I20260811 12:38:34.775259 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3fafa0a7ec541e9a42c826e4d0a821d" candidate_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb807c78ecb4eb689307f2929f61db3"
I20260811 12:38:34.775323 6026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3fafa0a7ec541e9a42c826e4d0a821d" candidate_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "919722c1774c4f8dad480c70659fb548"
I20260811 12:38:34.775512 6260 raft_consensus.cc:3037] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:34.775578 6026 raft_consensus.cc:3037] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:34.778918 6260 raft_consensus.cc:2454] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20fadd2aa46c4ad8bcd401b807ae7ba6 in term 1.
I20260811 12:38:34.778918 6026 raft_consensus.cc:2454] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20fadd2aa46c4ad8bcd401b807ae7ba6 in term 1.
I20260811 12:38:34.779399 6386 leader_election.cc:302] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20fadd2aa46c4ad8bcd401b807ae7ba6, beb807c78ecb4eb689307f2929f61db3; no voters:
I20260811 12:38:34.779786 6634 raft_consensus.cc:2781] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:34.780305 6634 raft_consensus.cc:686] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 20fadd2aa46c4ad8bcd401b807ae7ba6, State: Running, Role: LEADER
I20260811 12:38:34.780815 6634 consensus_queue.cc:237] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } }
I20260811 12:38:34.786029 5757 catalog_manager.cc:5495] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20fadd2aa46c4ad8bcd401b807ae7ba6 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20fadd2aa46c4ad8bcd401b807ae7ba6" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43885 } health_report { overall_health: HEALTHY } } }
W20260811 12:38:34.789582 6578 tablet.cc:2369] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:38:34.808746 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.803493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.3.204.67:43885)
0811 12:38:34.803698 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.803704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.803723 (+ 19us) server_negotiation.cc:407] Connection header received
0811 12:38:34.803894 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.803897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.803980 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.804139 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:38:34.804517 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.805218 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.806603 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.806997 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.807081 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.807605 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:38:34.808162 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:38:34.808281 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.808396 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.808486 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:38:34.818050 6026 raft_consensus.cc:1261] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 1 FOLLOWER]: Refusing update from remote peer 20fadd2aa46c4ad8bcd401b807ae7ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:34.818050 6260 raft_consensus.cc:1261] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 1 FOLLOWER]: Refusing update from remote peer 20fadd2aa46c4ad8bcd401b807ae7ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:34.819211 6639 consensus_queue.cc:1035] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beb807c78ecb4eb689307f2929f61db3" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 38995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:34.819589 6634 consensus_queue.cc:1035] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919722c1774c4f8dad480c70659fb548" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:34.847604 5758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:34.847949 5758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:34.851933 5758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2984d83ad13349ad94ee4892deca0342: Sending DeleteTablet for 3 replicas of tablet c3fafa0a7ec541e9a42c826e4d0a821d
I20260811 12:38:34.853176 5994 tablet_service.cc:1514] Processing DeleteTablet for tablet c3fafa0a7ec541e9a42c826e4d0a821d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:34 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:38314
I20260811 12:38:34.853215 6222 tablet_service.cc:1514] Processing DeleteTablet for tablet c3fafa0a7ec541e9a42c826e4d0a821d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:34 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:40630
I20260811 12:38:34.853432 6451 tablet_service.cc:1514] Processing DeleteTablet for tablet c3fafa0a7ec541e9a42c826e4d0a821d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:34 UTC) from {username='kudu', principal='kudu/127.3.204.126@KRBTEST.COM'} at 127.0.0.1:51594
I20260811 12:38:34.855381 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 5874
I20260811 12:38:34.872524 6658 tablet_replica.cc:330] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: stopping tablet replica
I20260811 12:38:34.873075 6661 tablet_replica.cc:330] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: stopping tablet replica
I20260811 12:38:34.873279 6658 raft_consensus.cc:2229] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:34.873594 6661 raft_consensus.cc:2229] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:34.874019 6658 raft_consensus.cc:2258] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:34.874151 6661 raft_consensus.cc:2258] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 11 12:38:34 dist-test-slave-lgr5 krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.65@KRBTEST.COM
I20260811 12:38:34.877581 6658 ts_tablet_manager.cc:1905] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:34.885226 6661 ts_tablet_manager.cc:1905] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:34.886014 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:34.856778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.3.204.65:43909)
0811 12:38:34.856973 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:34.856981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:34.864620 (+ 7639us) server_negotiation.cc:407] Connection header received
0811 12:38:34.864962 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:34.864970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:34.865070 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:34.865149 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:34.867237 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.868636 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.870790 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:34.871559 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:34.871681 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:34.877761 (+ 6080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:34.877778 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:34.877798 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:34.877849 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:34.881994 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:34.883044 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:34.883054 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:34.883062 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:34.883260 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:34.883773 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:34.883778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:34.883782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:34.884331 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:34.884619 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:34.885291 (+ 672us) server_negotiation.cc:299] Negotiation successful
0811 12:38:34.885541 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:38:34.887481 6091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:34.887676 6666 tablet_replica.cc:330] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: stopping tablet replica
I20260811 12:38:34.891443 6661 ts_tablet_manager.cc:1918] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:34.891635 6661 log.cc:1196] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/c3fafa0a7ec541e9a42c826e4d0a821d
I20260811 12:38:34.892134 6661 ts_tablet_manager.cc:1939] T c3fafa0a7ec541e9a42c826e4d0a821d P beb807c78ecb4eb689307f2929f61db3: Deleting consensus metadata
I20260811 12:38:34.892318 6666 raft_consensus.cc:2229] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:38:34.893256 6666 raft_consensus.cc:2258] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:34.894506 5736 catalog_manager.cc:4841] TS beb807c78ecb4eb689307f2929f61db3 (127.3.204.66:38995): tablet c3fafa0a7ec541e9a42c826e4d0a821d (table test-table [id=2793f1ede00f48289b63634fbb2e5422]) successfully deleted
I20260811 12:38:34.895836 6666 ts_tablet_manager.cc:1905] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:34.904448 6666 ts_tablet_manager.cc:1918] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:34.904623 6666 log.cc:1196] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/c3fafa0a7ec541e9a42c826e4d0a821d
I20260811 12:38:34.905113 6666 ts_tablet_manager.cc:1939] T c3fafa0a7ec541e9a42c826e4d0a821d P 20fadd2aa46c4ad8bcd401b807ae7ba6: Deleting consensus metadata
I20260811 12:38:34.906723 5730 catalog_manager.cc:4841] TS 20fadd2aa46c4ad8bcd401b807ae7ba6 (127.3.204.67:43885): tablet c3fafa0a7ec541e9a42c826e4d0a821d (table test-table [id=2793f1ede00f48289b63634fbb2e5422]) successfully deleted
I20260811 12:38:35.016453 6658 ts_tablet_manager.cc:1918] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:35.016729 6658 log.cc:1196] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/c3fafa0a7ec541e9a42c826e4d0a821d
I20260811 12:38:35.017370 6658 ts_tablet_manager.cc:1939] T c3fafa0a7ec541e9a42c826e4d0a821d P 919722c1774c4f8dad480c70659fb548: Deleting consensus metadata
I20260811 12:38:35.017419 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 5874
I20260811 12:38:35.046191 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6110
W20260811 12:38:35.046193 5735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.204.65:43909 (error 108)
I20260811 12:38:35.048607 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.047926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38326)
0811 12:38:35.048300 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:35.048422 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
W20260811 12:38:35.048897 5735 catalog_manager.cc:4584] TS 919722c1774c4f8dad480c70659fb548 (127.3.204.65:43909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3fafa0a7ec541e9a42c826e4d0a821d: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Aug 11 12:38:35 dist-test-slave-lgr5 krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.66@KRBTEST.COM
I20260811 12:38:35.064597 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.047371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.3.204.66:38995)
0811 12:38:35.047571 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:35.047577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:35.047599 (+ 22us) server_negotiation.cc:407] Connection header received
0811 12:38:35.047748 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:35.047754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:35.047838 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:35.047935 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:35.049753 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:35.050893 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:35.052354 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:35.053017 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:35.053174 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:35.057445 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:35.057500 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:35.057523 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:35.057615 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:35.061350 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:35.062166 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:35.062176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:35.062180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:35.062356 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:35.062717 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:35.062721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:35.062725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:35.063252 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:35.063568 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:35.063890 (+ 322us) server_negotiation.cc:299] Negotiation successful
0811 12:38:35.064173 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:38:35.065387 6334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:35.102782 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.102261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38336)
0811 12:38:35.102516 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:35.102622 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260811 12:38:35.103863 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.103496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38352)
0811 12:38:35.103649 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:35.103739 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 12:38:35.164717 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.164173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38356)
0811 12:38:35.164393 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:35.164505 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260811 12:38:35.166174 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.165755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38360)
0811 12:38:35.165917 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:35.166019 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 12:38:35.198680 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 6110
I20260811 12:38:35.223114 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6354
Aug 11 12:38:35 dist-test-slave-lgr5 krb5kdc[5682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.67@KRBTEST.COM
I20260811 12:38:35.241995 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.224012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.3.204.67:43885)
0811 12:38:35.224271 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:35.224278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:35.224305 (+ 27us) server_negotiation.cc:407] Connection header received
0811 12:38:35.224421 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:35.224427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:35.224507 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:35.224610 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:35.226023 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:35.227215 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:35.228867 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:35.229596 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:35.229739 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:35.234515 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:35.234530 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:35.234548 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:35.234595 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:35.238419 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:35.239430 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:35.239441 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:35.239445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:35.239630 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:35.240114 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:35.240121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:35.240124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:35.240655 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:35.241003 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:35.241315 (+ 312us) server_negotiation.cc:299] Negotiation successful
0811 12:38:35.241578 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260811 12:38:35.242874 6568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:35.246840 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.246221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38376)
0811 12:38:35.246470 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:35.246603 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 12:38:35.248031 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.247601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38382)
0811 12:38:35.247795 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:35.247879 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260811 12:38:35.370054 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 6354
I20260811 12:38:35.401777 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 5705
I20260811 12:38:35.403009 5823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:35.514526 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:35.514041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:43909 (local address 127.0.0.1:38394)
0811 12:38:35.514256 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:35.514341 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.65:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 12:38:35.514832 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 5705
2026-08-11T12:38:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4788 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-11T12:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:35Z Disabled control of system clock
I20260811 12:38:35.611702 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:39147
--webserver_interface=127.3.204.126
--webserver_port=0
--builtin_ntp_servers=127.3.204.84:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:39147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 12:38:35.958853 6683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:35.959164 6683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:35.959290 6683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:35.966660 6683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:35.966780 6683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:35.966853 6683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:35.966926 6683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:35.966979 6683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:35.977174 6683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:39147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:39147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:35.978703 6683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:35.980579 6683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:35.992157 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:35.992362 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:35.992352 6692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:35.993337 6683 server_base.cc:1034] running on GCE node
I20260811 12:38:35.994148 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:35.995636 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:35.996832 6683 hybrid_clock.cc:648] HybridClock initialized: now 1786451915996782 us; error 40 us; skew 500 ppm
I20260811 12:38:35.997191 6683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:35.999542 6683 webserver.cc:460] Webserver started at http://127.3.204.126:37943/ using document root <none> and password file <none>
I20260811 12:38:36.000330 6683 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:36.000447 6683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:36.000778 6683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:36.003886 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "cabe58c9fc30492cb54b5f910dd160ef"
format_stamp: "Formatted at 2026-08-11 12:38:35 on dist-test-slave-lgr5"
server_key: "aad594b866f3f2b88b0b122721c1c778"
server_key_iv: "4e2fea244ee5f797bdb46ff822cdd4de"
server_key_version: "encryptionkey@0"
I20260811 12:38:36.004777 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "cabe58c9fc30492cb54b5f910dd160ef"
format_stamp: "Formatted at 2026-08-11 12:38:35 on dist-test-slave-lgr5"
server_key: "aad594b866f3f2b88b0b122721c1c778"
server_key_iv: "4e2fea244ee5f797bdb46ff822cdd4de"
server_key_version: "encryptionkey@0"
I20260811 12:38:36.010283 6683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260811 12:38:36.013911 6699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:36.015327 6683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 12:38:36.015498 6683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "cabe58c9fc30492cb54b5f910dd160ef"
format_stamp: "Formatted at 2026-08-11 12:38:35 on dist-test-slave-lgr5"
server_key: "aad594b866f3f2b88b0b122721c1c778"
server_key_iv: "4e2fea244ee5f797bdb46ff822cdd4de"
server_key_version: "encryptionkey@0"
I20260811 12:38:36.015707 6683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:36.026719 6683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:36.027555 6683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:36.027809 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:36.044742 6683 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:39147
I20260811 12:38:36.044766 6787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:39147 every 8 connection(s)
I20260811 12:38:36.046115 6683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:36.049978 6789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:36.051797 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 6683
I20260811 12:38:36.051944 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:36.052438 3889 external_mini_cluster.cc:1428] Setting key 80ffbe924cd9d892a121380d0bebed52
I20260811 12:38:36.061838 6789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef: Bootstrap starting.
I20260811 12:38:36.065291 6789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:36.066323 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:36.054442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.3.204.126:39147)
0811 12:38:36.055499 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:36.055511 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:36.055559 (+ 48us) server_negotiation.cc:407] Connection header received
0811 12:38:36.057316 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:36.057342 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:36.057815 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:36.058164 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:36.058768 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:36.060196 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:36.061553 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:36.062269 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:36.062460 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:36.063710 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:36.063719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:36.063734 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:36.064399 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:36.064495 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:36.064524 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:36.064644 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:36.064888 (+ 244us) server_negotiation.cc:299] Negotiation successful
0811 12:38:36.065227 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":347,"threads_started":1}
I20260811 12:38:36.066403 6789 log.cc:824] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:36.069841 6789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef: No bootstrap required, opened a new log
I20260811 12:38:36.077773 6789 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabe58c9fc30492cb54b5f910dd160ef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39147 } }
I20260811 12:38:36.078213 6789 raft_consensus.cc:374] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:36.078311 6789 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabe58c9fc30492cb54b5f910dd160ef, State: Initialized, Role: FOLLOWER
I20260811 12:38:36.078917 6789 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabe58c9fc30492cb54b5f910dd160ef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39147 } }
I20260811 12:38:36.079156 6789 raft_consensus.cc:388] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:36.079285 6789 raft_consensus.cc:482] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:36.079449 6789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:36.081753 6789 raft_consensus.cc:504] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabe58c9fc30492cb54b5f910dd160ef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39147 } }
I20260811 12:38:36.082283 6789 leader_election.cc:302] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cabe58c9fc30492cb54b5f910dd160ef; no voters:
I20260811 12:38:36.082906 6789 leader_election.cc:288] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:36.083070 6801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:36.083794 6801 raft_consensus.cc:686] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [term 1 LEADER]: Becoming Leader. State: Replica: cabe58c9fc30492cb54b5f910dd160ef, State: Running, Role: LEADER
I20260811 12:38:36.084410 6801 consensus_queue.cc:237] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabe58c9fc30492cb54b5f910dd160ef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39147 } }
I20260811 12:38:36.085106 6789 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:36.087348 6803 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cabe58c9fc30492cb54b5f910dd160ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabe58c9fc30492cb54b5f910dd160ef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39147 } } }
I20260811 12:38:36.087817 6803 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:36.087458 6804 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabe58c9fc30492cb54b5f910dd160ef. Latest consensus state: current_term: 1 leader_uuid: "cabe58c9fc30492cb54b5f910dd160ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabe58c9fc30492cb54b5f910dd160ef" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 39147 } } }
I20260811 12:38:36.088166 6804 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:36.089636 6811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:36.094326 6811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:36.102802 6811 catalog_manager.cc:1347] Generated new cluster ID: 85d125bd96b34fe0baebde60e92855f8
I20260811 12:38:36.102926 6811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:36.112378 6811 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:36.113688 6811 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:36.123342 6811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef: Generated new TSK 0
I20260811 12:38:36.124315 6811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:36.136503 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.65:0
--local_ip_for_outbound_sockets=127.3.204.65
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39147
--builtin_ntp_servers=127.3.204.84:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 12:38:36.482648 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:36.482960 6827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:36.483081 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:36.490602 6827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:36.490712 6827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:36.490861 6827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.65
I20260811 12:38:36.500497 6827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.204.65
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.65
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:36.501853 6827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:36.503713 6827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:36.516908 6834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:36.516906 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:36.517141 6840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:36.517871 6827 server_base.cc:1034] running on GCE node
I20260811 12:38:36.518491 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:36.519455 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:36.520649 6827 hybrid_clock.cc:648] HybridClock initialized: now 1786451916520594 us; error 43 us; skew 500 ppm
I20260811 12:38:36.520968 6827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:36.523186 6827 webserver.cc:460] Webserver started at http://127.3.204.65:32865/ using document root <none> and password file <none>
I20260811 12:38:36.523875 6827 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:36.523977 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:36.524330 6827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:36.527247 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/data/instance:
uuid: "75859adb8133465aa8a6d9a687342641"
format_stamp: "Formatted at 2026-08-11 12:38:36 on dist-test-slave-lgr5"
server_key: "e60b95549577315bcfef1d40bc283050"
server_key_iv: "aa2a5c2c51711029b226912bfa9e7a5c"
server_key_version: "encryptionkey@0"
I20260811 12:38:36.528070 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance:
uuid: "75859adb8133465aa8a6d9a687342641"
format_stamp: "Formatted at 2026-08-11 12:38:36 on dist-test-slave-lgr5"
server_key: "e60b95549577315bcfef1d40bc283050"
server_key_iv: "aa2a5c2c51711029b226912bfa9e7a5c"
server_key_version: "encryptionkey@0"
I20260811 12:38:36.533480 6827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260811 12:38:36.537071 6848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:36.538380 6827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 12:38:36.538533 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/wal
uuid: "75859adb8133465aa8a6d9a687342641"
format_stamp: "Formatted at 2026-08-11 12:38:36 on dist-test-slave-lgr5"
server_key: "e60b95549577315bcfef1d40bc283050"
server_key_iv: "aa2a5c2c51711029b226912bfa9e7a5c"
server_key_version: "encryptionkey@0"
I20260811 12:38:36.538741 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:36.561738 6827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:36.562623 6827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:36.562929 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:36.564028 6827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:36.565565 6827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:36.565665 6827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:36.565770 6827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:36.565840 6827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:36.597432 6827 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.65:38591
I20260811 12:38:36.597512 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.65:38591 every 8 connection(s)
I20260811 12:38:36.598929 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/data/info.pb
I20260811 12:38:36.608372 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 6827
I20260811 12:38:36.608541 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/wal/instance
I20260811 12:38:36.608982 3889 external_mini_cluster.cc:1428] Setting key cc21bf7ebf5d1b71e5c5376a96021a7a
I20260811 12:38:36.611521 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:36.601555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.65:51969 (local address 127.3.204.126:39147)
0811 12:38:36.601981 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:36.601985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:36.603235 (+ 1250us) server_negotiation.cc:407] Connection header received
0811 12:38:36.605097 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:36.605103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:36.605185 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:36.605286 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:36.606380 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:36.607126 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:36.608730 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:36.609148 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:36.609258 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:36.610697 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:36.610701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:36.610704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:36.610723 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:36.610765 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:36.610776 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:36.610852 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:36.611070 (+ 218us) server_negotiation.cc:299] Negotiation successful
0811 12:38:36.611167 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":217,"threads_started":1}
I20260811 12:38:36.611713 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.66:0
--local_ip_for_outbound_sockets=127.3.204.66
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39147
--builtin_ntp_servers=127.3.204.84:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 12:38:36.612617 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:36.602150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:39147 (local address 127.3.204.65:51969)
0811 12:38:36.603059 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:36.603106 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:36.604815 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:36.605532 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:36.605540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:36.605635 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:36.606215 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:36.606232 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:36.607280 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:36.607284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:36.608593 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:36.608610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:36.609377 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:36.609385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:36.609698 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:36.610393 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:36.610396 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:36.610436 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:36.610521 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:36.610865 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:36.610876 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:36.610885 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 12:38:36.611048 (+ 163us) client_negotiation.cc:240] Negotiation successful
0811 12:38:36.611438 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":265,"threads_started":1}
I20260811 12:38:36.614111 7034 heartbeater.cc:347] Connected to a master server at 127.3.204.126:39147
I20260811 12:38:36.614399 7034 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:36.615147 7034 heartbeater.cc:511] Master 127.3.204.126:39147 requested a full tablet report, sending...
I20260811 12:38:36.617318 6731 ts_manager.cc:194] Registered new tserver with Master: 75859adb8133465aa8a6d9a687342641 (127.3.204.65:38591)
I20260811 12:38:36.619269 6731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.65:51969
W20260811 12:38:36.957799 7040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:36.958107 7040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:36.958192 7040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:36.965482 7040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:36.965595 7040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:36.965710 7040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.66
I20260811 12:38:36.975072 7040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.204.66
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.66
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:36.976444 7040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:36.978300 7040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:36.991870 7055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:36.991907 7049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:36.991935 7051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:36.992705 7040 server_base.cc:1034] running on GCE node
I20260811 12:38:36.993386 7040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:36.994429 7040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:36.995653 7040 hybrid_clock.cc:648] HybridClock initialized: now 1786451916995592 us; error 51 us; skew 500 ppm
I20260811 12:38:36.996011 7040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:36.998394 7040 webserver.cc:460] Webserver started at http://127.3.204.66:46339/ using document root <none> and password file <none>
I20260811 12:38:36.999135 7040 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:36.999243 7040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:36.999569 7040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:37.002686 7040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/data/instance:
uuid: "9e2efe1f3d7a427b9b18a3426d59e925"
format_stamp: "Formatted at 2026-08-11 12:38:36 on dist-test-slave-lgr5"
server_key: "3583bd0a87b3634aa1c0fe7c3948dc53"
server_key_iv: "168e9bbbfa1db4e8a847b9390ff0e9c6"
server_key_version: "encryptionkey@0"
I20260811 12:38:37.003577 7040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance:
uuid: "9e2efe1f3d7a427b9b18a3426d59e925"
format_stamp: "Formatted at 2026-08-11 12:38:36 on dist-test-slave-lgr5"
server_key: "3583bd0a87b3634aa1c0fe7c3948dc53"
server_key_iv: "168e9bbbfa1db4e8a847b9390ff0e9c6"
server_key_version: "encryptionkey@0"
I20260811 12:38:37.009209 7040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260811 12:38:37.012856 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:37.014283 7040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:37.014462 7040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/wal
uuid: "9e2efe1f3d7a427b9b18a3426d59e925"
format_stamp: "Formatted at 2026-08-11 12:38:36 on dist-test-slave-lgr5"
server_key: "3583bd0a87b3634aa1c0fe7c3948dc53"
server_key_iv: "168e9bbbfa1db4e8a847b9390ff0e9c6"
server_key_version: "encryptionkey@0"
I20260811 12:38:37.014680 7040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:37.032963 7040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:37.033835 7040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:37.034134 7040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:37.035197 7040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:37.036859 7040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:37.036963 7040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:37.037075 7040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:37.037145 7040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:37.069593 7040 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.66:35291
I20260811 12:38:37.069648 7257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.66:35291 every 8 connection(s)
I20260811 12:38:37.070973 7040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/data/info.pb
I20260811 12:38:37.074594 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7040
I20260811 12:38:37.074775 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/wal/instance
I20260811 12:38:37.075219 3889 external_mini_cluster.cc:1428] Setting key 1fa99720ad9949608bead4561362f679
I20260811 12:38:37.078191 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.204.67:0
--local_ip_for_outbound_sockets=127.3.204.67
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39147
--builtin_ntp_servers=127.3.204.84:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 12:38:37.084735 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.073570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.66:42065 (local address 127.3.204.126:39147)
0811 12:38:37.073746 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.073750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.075367 (+ 1617us) server_negotiation.cc:407] Connection header received
0811 12:38:37.077412 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.077417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.077510 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.077628 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:37.078753 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.079748 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.081549 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.082182 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.082317 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.083847 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:37.083853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:37.083856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:37.083878 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:37.083913 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:37.083925 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:37.084006 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.084283 (+ 277us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.084412 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 12:38:37.085884 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.074079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:39147 (local address 127.3.204.66:42065)
0811 12:38:37.075182 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:37.075229 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:37.077138 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:37.077813 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:37.077821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:37.077934 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:37.078574 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.078597 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.079950 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.079954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.081378 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.081396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.082446 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.082455 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.082699 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.083435 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:37.083437 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:37.083506 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:37.083620 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:37.084019 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:37.084032 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:37.084041 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 12:38:37.084253 (+ 212us) client_negotiation.cc:240] Negotiation successful
0811 12:38:37.084717 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":328,"threads_started":1}
I20260811 12:38:37.087450 7258 heartbeater.cc:347] Connected to a master server at 127.3.204.126:39147
I20260811 12:38:37.087769 7258 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:37.088690 7258 heartbeater.cc:511] Master 127.3.204.126:39147 requested a full tablet report, sending...
I20260811 12:38:37.090286 6731 ts_manager.cc:194] Registered new tserver with Master: 9e2efe1f3d7a427b9b18a3426d59e925 (127.3.204.66:35291)
I20260811 12:38:37.091351 6731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.66:42065
W20260811 12:38:37.428812 7261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:37.429131 7261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:37.429265 7261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:37.436686 7261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:37.436816 7261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:37.436964 7261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.204.67
I20260811 12:38:37.446554 7261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.204.67
--webserver_port=0
--tserver_master_addrs=127.3.204.126:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.204.67
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:37.447908 7261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:37.449769 7261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:37.462815 7268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:37.462774 7272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:37.462793 7269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:37.463431 7261 server_base.cc:1034] running on GCE node
I20260811 12:38:37.464107 7261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:37.465112 7261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:37.466339 7261 hybrid_clock.cc:648] HybridClock initialized: now 1786451917466282 us; error 47 us; skew 500 ppm
I20260811 12:38:37.466699 7261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 12:38:37.469038 7261 webserver.cc:460] Webserver started at http://127.3.204.67:36177/ using document root <none> and password file <none>
I20260811 12:38:37.469741 7261 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:37.469846 7261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:37.470161 7261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:37.473248 7261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/data/instance:
uuid: "e729c3bc122b4896a51ef1fb05120064"
format_stamp: "Formatted at 2026-08-11 12:38:37 on dist-test-slave-lgr5"
server_key: "4ad2f94ff4035511bb001e451988e80c"
server_key_iv: "6025f3bb4d19d02ffa0f096189d557bc"
server_key_version: "encryptionkey@0"
I20260811 12:38:37.474057 7261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance:
uuid: "e729c3bc122b4896a51ef1fb05120064"
format_stamp: "Formatted at 2026-08-11 12:38:37 on dist-test-slave-lgr5"
server_key: "4ad2f94ff4035511bb001e451988e80c"
server_key_iv: "6025f3bb4d19d02ffa0f096189d557bc"
server_key_version: "encryptionkey@0"
I20260811 12:38:37.479334 7261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260811 12:38:37.482896 7285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:37.484293 7261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:38:37.484452 7261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/wal
uuid: "e729c3bc122b4896a51ef1fb05120064"
format_stamp: "Formatted at 2026-08-11 12:38:37 on dist-test-slave-lgr5"
server_key: "4ad2f94ff4035511bb001e451988e80c"
server_key_iv: "6025f3bb4d19d02ffa0f096189d557bc"
server_key_version: "encryptionkey@0"
I20260811 12:38:37.484663 7261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:37.515278 7261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:37.516232 7261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:37.516535 7261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:37.517541 7261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 12:38:37.519089 7261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 12:38:37.519192 7261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:37.519299 7261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 12:38:37.519374 7261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:37.550542 7261 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.67:43793
I20260811 12:38:37.550624 7488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.67:43793 every 8 connection(s)
I20260811 12:38:37.551867 7261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/data/info.pb
I20260811 12:38:37.553525 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7261
I20260811 12:38:37.553685 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/wal/instance
I20260811 12:38:37.554221 3889 external_mini_cluster.cc:1428] Setting key 60f8d365de297f3b912a346f33a2c226
I20260811 12:38:37.564663 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.554554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:48433 (local address 127.3.204.126:39147)
0811 12:38:37.554733 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.554739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.556433 (+ 1694us) server_negotiation.cc:407] Connection header received
0811 12:38:37.558193 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.558197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.558262 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.558343 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:37.559450 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.560264 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.561864 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.562286 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.562376 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.563727 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:37.563732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:37.563736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:37.563756 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:37.563790 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:37.563802 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:37.563886 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.564198 (+ 312us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.564312 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 12:38:37.565739 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.555164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:39147 (local address 127.3.204.67:48433)
0811 12:38:37.556271 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:37.556326 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:37.557957 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:37.558576 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:37.558585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:37.558676 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:38:37.559282 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.559300 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.560426 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.560430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.561741 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.561760 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.562478 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.562484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.562688 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.563368 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 12:38:37.563370 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:38:37.563412 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 12:38:37.563519 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:38:37.563903 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:38:37.563916 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:38:37.563923 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 12:38:37.564174 (+ 251us) client_negotiation.cc:240] Negotiation successful
0811 12:38:37.564594 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":337,"threads_started":1}
I20260811 12:38:37.567090 7491 heartbeater.cc:347] Connected to a master server at 127.3.204.126:39147
I20260811 12:38:37.567431 7491 heartbeater.cc:464] Registering TS with master...
I20260811 12:38:37.568460 7491 heartbeater.cc:511] Master 127.3.204.126:39147 requested a full tablet report, sending...
I20260811 12:38:37.569968 6731 ts_manager.cc:194] Registered new tserver with Master: e729c3bc122b4896a51ef1fb05120064 (127.3.204.67:43793)
I20260811 12:38:37.570755 6731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.204.67:48433
I20260811 12:38:37.581773 3889 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 12:38:37.592070 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.585927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.3.204.126:39147)
0811 12:38:37.586185 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.586191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.586605 (+ 414us) server_negotiation.cc:407] Connection header received
0811 12:38:37.586772 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.586778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.586873 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.586994 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:37.587542 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.588398 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.589959 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.590377 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.590476 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.591418 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:37.591424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:37.591427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:37.591445 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:37.591472 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:37.591484 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:37.591551 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.591705 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.591780 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 12:38:37.595774 6731 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 12:38:37.605248 6731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 12:38:37.622349 7034 heartbeater.cc:503] Master 127.3.204.126:39147 was elected leader, sending a full tablet report...
I20260811 12:38:37.627992 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.618539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:35291 (local address 127.0.0.1:59526)
0811 12:38:37.619108 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:37.619124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:37.619225 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:37.620393 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:37.620399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:37.620430 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:37.620882 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.620899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.623048 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.623055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.625641 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.625656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.626961 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.626969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.627187 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.627240 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 12:38:37.627362 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 12:38:37.627605 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":45,"thread_start_us":709,"threads_started":1}
I20260811 12:38:37.628175 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.619053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.3.204.66:35291)
0811 12:38:37.619735 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.619746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.619773 (+ 27us) server_negotiation.cc:407] Connection header received
0811 12:38:37.619869 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.619876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.620135 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.621153 (+ 1018us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:37.621390 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.622894 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.625831 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.626763 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.626887 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.627148 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.627642 (+ 494us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.627816 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":424,"threads_started":1}
I20260811 12:38:37.629840 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.619721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.67:43793 (local address 127.0.0.1:53634)
0811 12:38:37.621638 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:37.621656 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:37.621766 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:37.622827 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:37.622832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:37.622870 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:37.623338 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.623355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.625517 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.625523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.627538 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.627556 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.629154 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.629160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.629331 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.629363 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 12:38:37.629423 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 12:38:37.629541 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":499,"threads_started":1}
I20260811 12:38:37.629930 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.617954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.3.204.65:38591)
0811 12:38:37.618846 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.618855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.618894 (+ 39us) server_negotiation.cc:407] Connection header received
0811 12:38:37.619373 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.619382 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.619650 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.619851 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:37.621260 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.623313 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.625772 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.627201 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.627323 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.627588 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.629426 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.629590 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":578,"threads_started":1}
I20260811 12:38:37.630492 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.617642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38591 (local address 127.0.0.1:46734)
0811 12:38:37.618745 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:37.618802 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:37.619011 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:37.619883 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:37.619890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:37.619922 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:37.621020 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.621040 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.623512 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.623517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.625543 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.625562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.627395 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.627401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.628991 (+ 1590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.629027 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 12:38:37.629108 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 12:38:37.630171 (+ 1063us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":1377,"thread_start_us":1290,"threads_started":1}
I20260811 12:38:37.631841 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.620356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.3.204.67:43793)
0811 12:38:37.622308 (+ 1952us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.622317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.622350 (+ 33us) server_negotiation.cc:407] Connection header received
0811 12:38:37.622439 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.622446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.622640 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.623558 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:37.623824 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.625376 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.627719 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.628932 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.630208 (+ 1276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.630466 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.630539 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.631490 (+ 951us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1782,"thread_start_us":1630,"threads_started":1}
I20260811 12:38:37.631681 7153 tablet_service.cc:1467] Processing CreateTablet for tablet 717192c204984a43bb79f37284b9d087 (DEFAULT_TABLE table=test-table [id=0c7603a364f441ad8fa99d7f36dc204a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:37.633057 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717192c204984a43bb79f37284b9d087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:37.633949 6929 tablet_service.cc:1467] Processing CreateTablet for tablet 717192c204984a43bb79f37284b9d087 (DEFAULT_TABLE table=test-table [id=0c7603a364f441ad8fa99d7f36dc204a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:37.635262 6929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717192c204984a43bb79f37284b9d087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:37.635380 7367 tablet_service.cc:1467] Processing CreateTablet for tablet 717192c204984a43bb79f37284b9d087 (DEFAULT_TABLE table=test-table [id=0c7603a364f441ad8fa99d7f36dc204a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 12:38:37.636746 7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717192c204984a43bb79f37284b9d087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:37.647032 7519 tablet_bootstrap.cc:492] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: Bootstrap starting.
I20260811 12:38:37.648068 7521 tablet_bootstrap.cc:492] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: Bootstrap starting.
I20260811 12:38:37.651500 7523 tablet_bootstrap.cc:492] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: Bootstrap starting.
I20260811 12:38:37.651942 7521 tablet_bootstrap.cc:654] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:37.652105 7519 tablet_bootstrap.cc:654] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:37.653328 7521 log.cc:824] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:37.653337 7519 log.cc:824] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:37.654618 7523 tablet_bootstrap.cc:654] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:37.655635 7523 log.cc:824] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:37.656646 7521 tablet_bootstrap.cc:492] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: No bootstrap required, opened a new log
I20260811 12:38:37.656646 7519 tablet_bootstrap.cc:492] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: No bootstrap required, opened a new log
I20260811 12:38:37.656944 7519 ts_tablet_manager.cc:1397] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260811 12:38:37.656944 7521 ts_tablet_manager.cc:1397] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 12:38:37.658438 7523 tablet_bootstrap.cc:492] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: No bootstrap required, opened a new log
I20260811 12:38:37.658774 7523 ts_tablet_manager.cc:1397] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260811 12:38:37.666131 7523 raft_consensus.cc:348] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.666533 7523 raft_consensus.cc:374] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:37.666626 7523 raft_consensus.cc:729] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e729c3bc122b4896a51ef1fb05120064, State: Initialized, Role: FOLLOWER
I20260811 12:38:37.667244 7523 consensus_queue.cc:260] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.667529 7519 raft_consensus.cc:348] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.667524 7521 raft_consensus.cc:348] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.667974 7521 raft_consensus.cc:374] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:37.667973 7519 raft_consensus.cc:374] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:37.668100 7519 raft_consensus.cc:729] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75859adb8133465aa8a6d9a687342641, State: Initialized, Role: FOLLOWER
I20260811 12:38:37.668100 7521 raft_consensus.cc:729] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2efe1f3d7a427b9b18a3426d59e925, State: Initialized, Role: FOLLOWER
I20260811 12:38:37.668562 7491 heartbeater.cc:503] Master 127.3.204.126:39147 was elected leader, sending a full tablet report...
I20260811 12:38:37.668757 7519 consensus_queue.cc:260] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.668877 7521 consensus_queue.cc:260] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.669088 7523 ts_tablet_manager.cc:1428] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260811 12:38:37.671221 7258 heartbeater.cc:503] Master 127.3.204.126:39147 was elected leader, sending a full tablet report...
I20260811 12:38:37.671926 7519 ts_tablet_manager.cc:1428] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260811 12:38:37.673390 7521 ts_tablet_manager.cc:1428] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
I20260811 12:38:37.686522 7533 raft_consensus.cc:482] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 12:38:37.686825 7533 raft_consensus.cc:504] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.688747 7533 leader_election.cc:288] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75859adb8133465aa8a6d9a687342641 (127.3.204.65:38591), 9e2efe1f3d7a427b9b18a3426d59e925 (127.3.204.66:35291)
I20260811 12:38:37.694386 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.689220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:55985 (local address 127.3.204.65:38591)
0811 12:38:37.689403 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.689408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.689429 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:38:37.689657 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.689663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.689756 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.689847 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:37.690338 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.691049 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.692771 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.693545 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.693693 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.693768 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.694004 (+ 236us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.694141 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 12:38:37.694406 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.688979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.65:38591 (local address 127.3.204.67:55985)
0811 12:38:37.689278 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:37.689298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:37.689450 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:37.689928 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:37.689934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:37.689967 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:37.690133 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.690141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.691191 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.691195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.692602 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.692616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.693692 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.693698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.693858 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.693882 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 12:38:37.693962 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 12:38:37.694103 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1}
I20260811 12:38:37.695164 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.689368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.204.67:59223 (local address 127.3.204.66:35291)
0811 12:38:37.689589 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.689595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.690181 (+ 586us) server_negotiation.cc:407] Connection header received
0811 12:38:37.690393 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.690397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.690468 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.690544 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 12:38:37.691058 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.691731 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.693447 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.694206 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.694350 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.694416 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.694828 (+ 412us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.694933 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 12:38:37.695187 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.689244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.66:35291 (local address 127.3.204.67:59223)
0811 12:38:37.690075 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0811 12:38:37.690091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:38:37.690210 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:38:37.690576 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:38:37.690581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:38:37.690605 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 12:38:37.690898 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.690906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.691872 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.691878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.693264 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:38:37.693277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.694366 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.694372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:38:37.694655 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.694684 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 12:38:37.694782 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 12:38:37.694902 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":442,"threads_started":1}
I20260811 12:38:37.695658 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717192c204984a43bb79f37284b9d087" candidate_uuid: "e729c3bc122b4896a51ef1fb05120064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75859adb8133465aa8a6d9a687342641" is_pre_election: true
I20260811 12:38:37.696218 6970 raft_consensus.cc:2454] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e729c3bc122b4896a51ef1fb05120064 in term 0.
I20260811 12:38:37.696146 7185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717192c204984a43bb79f37284b9d087" candidate_uuid: "e729c3bc122b4896a51ef1fb05120064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" is_pre_election: true
I20260811 12:38:37.696609 7185 raft_consensus.cc:2454] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e729c3bc122b4896a51ef1fb05120064 in term 0.
I20260811 12:38:37.696868 7292 leader_election.cc:302] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75859adb8133465aa8a6d9a687342641, e729c3bc122b4896a51ef1fb05120064; no voters:
I20260811 12:38:37.697260 7533 raft_consensus.cc:2781] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 12:38:37.697461 7533 raft_consensus.cc:482] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 12:38:37.697551 7533 raft_consensus.cc:3037] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:37.699692 7533 raft_consensus.cc:504] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.700361 7533 leader_election.cc:288] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [CANDIDATE]: Term 1 election: Requested vote from peers 75859adb8133465aa8a6d9a687342641 (127.3.204.65:38591), 9e2efe1f3d7a427b9b18a3426d59e925 (127.3.204.66:35291)
I20260811 12:38:37.700759 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717192c204984a43bb79f37284b9d087" candidate_uuid: "e729c3bc122b4896a51ef1fb05120064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75859adb8133465aa8a6d9a687342641"
I20260811 12:38:37.700841 7185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717192c204984a43bb79f37284b9d087" candidate_uuid: "e729c3bc122b4896a51ef1fb05120064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2efe1f3d7a427b9b18a3426d59e925"
I20260811 12:38:37.700942 6970 raft_consensus.cc:3037] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:37.701049 7185 raft_consensus.cc:3037] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:37.703150 6970 raft_consensus.cc:2454] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e729c3bc122b4896a51ef1fb05120064 in term 1.
I20260811 12:38:37.703235 7185 raft_consensus.cc:2454] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e729c3bc122b4896a51ef1fb05120064 in term 1.
I20260811 12:38:37.703557 7292 leader_election.cc:302] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75859adb8133465aa8a6d9a687342641, e729c3bc122b4896a51ef1fb05120064; no voters:
I20260811 12:38:37.703866 7533 raft_consensus.cc:2781] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:37.704334 7533 raft_consensus.cc:686] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 1 LEADER]: Becoming Leader. State: Replica: e729c3bc122b4896a51ef1fb05120064, State: Running, Role: LEADER
I20260811 12:38:37.704806 7533 consensus_queue.cc:237] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } }
I20260811 12:38:37.709664 6728 catalog_manager.cc:5495] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 reported cstate change: term changed from 0 to 1, leader changed from <none> to e729c3bc122b4896a51ef1fb05120064 (127.3.204.67). New cstate: current_term: 1 leader_uuid: "e729c3bc122b4896a51ef1fb05120064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e729c3bc122b4896a51ef1fb05120064" member_type: VOTER last_known_addr { host: "127.3.204.67" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 } health_report { overall_health: UNKNOWN } } }
I20260811 12:38:37.730971 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.725889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.3.204.67:43793)
0811 12:38:37.726063 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.726069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.726178 (+ 109us) server_negotiation.cc:407] Connection header received
0811 12:38:37.726352 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.726356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.726438 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.726583 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:38:37.726872 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.727637 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.729046 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.729445 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.729544 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.730026 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:38:37.730551 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:38:37.730667 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.730709 (+ 42us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.730774 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 12:38:37.739271 7185 raft_consensus.cc:1261] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 1 FOLLOWER]: Refusing update from remote peer e729c3bc122b4896a51ef1fb05120064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:37.739283 6970 raft_consensus.cc:1261] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 1 FOLLOWER]: Refusing update from remote peer e729c3bc122b4896a51ef1fb05120064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:38:37.740236 7546 consensus_queue.cc:1035] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e2efe1f3d7a427b9b18a3426d59e925" member_type: VOTER last_known_addr { host: "127.3.204.66" port: 35291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:37.740635 7533 consensus_queue.cc:1035] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75859adb8133465aa8a6d9a687342641" member_type: VOTER last_known_addr { host: "127.3.204.65" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:38:37.751950 7549 mvcc.cc:205] Tried to move back new op lower bound from 7317307055049969664 to 7317307054922698752. Current Snapshot: MvccSnapshot[applied={T|T < 7317307055049969664}]
I20260811 12:38:37.767232 6731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:37.767503 6731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 12:38:37.771013 6731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cabe58c9fc30492cb54b5f910dd160ef: Sending DeleteTablet for 3 replicas of tablet 717192c204984a43bb79f37284b9d087
I20260811 12:38:37.772300 7367 tablet_service.cc:1514] Processing DeleteTablet for tablet 717192c204984a43bb79f37284b9d087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:37 UTC) from {username='slave'} at 127.0.0.1:53634
I20260811 12:38:37.772656 7153 tablet_service.cc:1514] Processing DeleteTablet for tablet 717192c204984a43bb79f37284b9d087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:37 UTC) from {username='slave'} at 127.0.0.1:59526
I20260811 12:38:37.772692 6929 tablet_service.cc:1514] Processing DeleteTablet for tablet 717192c204984a43bb79f37284b9d087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 12:38:37 UTC) from {username='slave'} at 127.0.0.1:46734
I20260811 12:38:37.773849 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6827
I20260811 12:38:37.787323 7571 tablet_replica.cc:330] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: stopping tablet replica
I20260811 12:38:37.788035 7571 raft_consensus.cc:2229] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:37.789474 7568 tablet_replica.cc:330] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: stopping tablet replica
I20260811 12:38:37.789785 7571 raft_consensus.cc:2258] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:37.790299 7568 raft_consensus.cc:2229] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 1 LEADER]: Raft consensus shutting down.
I20260811 12:38:37.792574 7571 ts_tablet_manager.cc:1905] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:37.793159 7568 raft_consensus.cc:2258] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:37.794387 7570 tablet_replica.cc:330] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: stopping tablet replica
I20260811 12:38:37.794852 7570 raft_consensus.cc:2229] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 12:38:37.795432 7570 raft_consensus.cc:2258] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 12:38:37.797812 7568 ts_tablet_manager.cc:1905] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:37.798504 7571 ts_tablet_manager.cc:1918] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:37.798655 7571 log.cc:1196] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-0/wal/wals/717192c204984a43bb79f37284b9d087
I20260811 12:38:37.799088 7571 ts_tablet_manager.cc:1939] T 717192c204984a43bb79f37284b9d087 P 75859adb8133465aa8a6d9a687342641: Deleting consensus metadata
I20260811 12:38:37.800760 6708 catalog_manager.cc:4841] TS 75859adb8133465aa8a6d9a687342641 (127.3.204.65:38591): tablet 717192c204984a43bb79f37284b9d087 (table test-table [id=0c7603a364f441ad8fa99d7f36dc204a]) successfully deleted
I20260811 12:38:37.802269 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.774907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.3.204.65:38591)
0811 12:38:37.775106 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.775113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.781751 (+ 6638us) server_negotiation.cc:407] Connection header received
0811 12:38:37.782704 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.782711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.782856 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.782948 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:37.784169 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.785347 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.790556 (+ 5209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.791247 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.791358 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.800656 (+ 9298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:37.800669 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:37.800686 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:37.801441 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:37.801504 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:37.801541 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:37.801652 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.801810 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.801930 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 12:38:37.803073 7029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:37.803750 7570 ts_tablet_manager.cc:1905] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 12:38:37.805487 7568 ts_tablet_manager.cc:1918] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:37.805670 7568 log.cc:1196] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-2/wal/wals/717192c204984a43bb79f37284b9d087
I20260811 12:38:37.806226 7568 ts_tablet_manager.cc:1939] T 717192c204984a43bb79f37284b9d087 P e729c3bc122b4896a51ef1fb05120064: Deleting consensus metadata
I20260811 12:38:37.808182 6708 catalog_manager.cc:4841] TS e729c3bc122b4896a51ef1fb05120064 (127.3.204.67:43793): tablet 717192c204984a43bb79f37284b9d087 (table test-table [id=0c7603a364f441ad8fa99d7f36dc204a]) successfully deleted
I20260811 12:38:37.810691 7570 ts_tablet_manager.cc:1918] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 12:38:37.810855 7570 log.cc:1196] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: Deleting WAL directory at /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786451585252647-3889-0/minicluster-data/ts-1/wal/wals/717192c204984a43bb79f37284b9d087
I20260811 12:38:37.811380 7570 ts_tablet_manager.cc:1939] T 717192c204984a43bb79f37284b9d087 P 9e2efe1f3d7a427b9b18a3426d59e925: Deleting consensus metadata
I20260811 12:38:37.813300 6703 catalog_manager.cc:4841] TS 9e2efe1f3d7a427b9b18a3426d59e925 (127.3.204.66:35291): tablet 717192c204984a43bb79f37284b9d087 (table test-table [id=0c7603a364f441ad8fa99d7f36dc204a]) successfully deleted
I20260811 12:38:37.895615 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 6827
I20260811 12:38:37.924278 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7040
I20260811 12:38:37.931151 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:37.925032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.3.204.66:35291)
0811 12:38:37.925207 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:37.925214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:37.925277 (+ 63us) server_negotiation.cc:407] Connection header received
0811 12:38:37.925470 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:37.925475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:37.925581 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:37.925674 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:37.926086 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.926829 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.928076 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:37.928592 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:37.928706 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:37.929806 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:37.929815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:37.929831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:37.930411 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:37.930464 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:37.930486 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:37.930598 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:37.930717 (+ 119us) server_negotiation.cc:299] Negotiation successful
0811 12:38:37.930832 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:38:37.931869 7249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:38.023061 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7040
I20260811 12:38:38.060276 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7261
I20260811 12:38:38.067492 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:38.061110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.3.204.67:43793)
0811 12:38:38.061305 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:38.061311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:38.061395 (+ 84us) server_negotiation.cc:407] Connection header received
0811 12:38:38.061613 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:38.061617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:38.061706 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:38.061799 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:38.062233 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:38.063025 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:38.064401 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:38.064850 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:38.064974 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:38.066060 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:38.066070 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:38.066087 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:38.066711 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:38.066763 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:38.066785 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:38.066881 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:38.067041 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 12:38:38.067149 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 12:38:38.068269 7479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:38.191188 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7261
I20260811 12:38:38.217626 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 6683
I20260811 12:38:38.218696 6779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:38.328110 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 6683
2026-08-11T12:38:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2815 ms)
[----------] 6 tests from EncryptionPolicyTest (22943 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:38 dist-test-slave-lgr5 krb5kdc[7585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:38 dist-test-slave-lgr5 krb5kdc[7585](info): set up 2 sockets
Aug 11 12:38:38 dist-test-slave-lgr5 krb5kdc[7585](info): commencing operation
krb5kdc: starting...
W20260811 12:38:39.629873 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:39 dist-test-slave-lgr5 krb5kdc[7585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:39.953331 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:40.573694 7607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:40.574231 7607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:40.574486 7607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:40.586639 7607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:40.586853 7607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:40.586987 7607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:40.587126 7607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:40.587231 7607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:40.604247 7607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39649
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:40.606582 7607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:40.609014 7607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:40.628569 7622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:40.628597 7615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:40.628655 7616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:40.630231 7607 server_base.cc:1034] running on GCE node
I20260811 12:38:40.631614 7607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:40.634073 7607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:40.635353 7607 hybrid_clock.cc:648] HybridClock initialized: now 1786451920635269 us; error 73 us; skew 500 ppm
I20260811 12:38:40.635977 7607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:40 dist-test-slave-lgr5 krb5kdc[7585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:40.641183 7607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:40.643807 7607 webserver.cc:460] Webserver started at http://127.0.0.1:45519/ using document root <none> and password file <none>
I20260811 12:38:40.645066 7607 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:40.645303 7607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:40.645790 7607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:40.651041 7607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "8e81729cb13143e18cbd0f9ea1a59587"
format_stamp: "Formatted at 2026-08-11 12:38:40 on dist-test-slave-lgr5"
server_key: "77fe36a347b1c8e4c1068246c1a5505e"
server_key_iv: "c27b6a7a23bcd41104a56009e5637d17"
server_key_version: "encryptionkey@0"
I20260811 12:38:40.652455 7607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "8e81729cb13143e18cbd0f9ea1a59587"
format_stamp: "Formatted at 2026-08-11 12:38:40 on dist-test-slave-lgr5"
server_key: "77fe36a347b1c8e4c1068246c1a5505e"
server_key_iv: "c27b6a7a23bcd41104a56009e5637d17"
server_key_version: "encryptionkey@0"
I20260811 12:38:40.661877 7607 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260811 12:38:40.668315 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:40.670843 7607 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260811 12:38:40.671208 7607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "8e81729cb13143e18cbd0f9ea1a59587"
format_stamp: "Formatted at 2026-08-11 12:38:40 on dist-test-slave-lgr5"
server_key: "77fe36a347b1c8e4c1068246c1a5505e"
server_key_iv: "c27b6a7a23bcd41104a56009e5637d17"
server_key_version: "encryptionkey@0"
I20260811 12:38:40.671557 7607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:40.698472 7607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:40.706324 7607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:40.706782 7607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:40.736791 7607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39649
I20260811 12:38:40.736851 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39649 every 8 connection(s)
I20260811 12:38:40.739166 7607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:40.740201 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7607
I20260811 12:38:40.740497 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:40.741703 3889 external_mini_cluster.cc:1428] Setting key 5dd41c896d9be2ceeb2ca86ceb8f7a74
I20260811 12:38:40.746202 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 12:38:40 dist-test-slave-lgr5 krb5kdc[7585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 12:38:40.769011 7724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587: Bootstrap starting.
I20260811 12:38:40.776997 7724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:40.779659 7724 log.cc:824] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:40.786067 7724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587: No bootstrap required, opened a new log
I20260811 12:38:40.786767 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:40.745162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36425 (local address 127.0.0.1:39649)
0811 12:38:40.746450 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:40.746491 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:40.746565 (+ 74us) server_negotiation.cc:407] Connection header received
0811 12:38:40.750515 (+ 3950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:40.750591 (+ 76us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:40.751780 (+ 1189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:40.752765 (+ 985us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:40.755138 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:40.758854 (+ 3716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:40.761188 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:40.763256 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:40.763578 (+ 322us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:40.769553 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:40.769569 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:40.769606 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:40.769723 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:40.777150 (+ 7427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:40.779008 (+ 1858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:40.779033 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:40.779061 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:40.779616 (+ 555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:40.780675 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:40.780692 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:40.780697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:40.781940 (+ 1243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:40.782515 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:40.783311 (+ 796us) server_negotiation.cc:299] Negotiation successful
0811 12:38:40.784068 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":516,"threads_started":1}
I20260811 12:38:40.794359 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:40.800858 7724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e81729cb13143e18cbd0f9ea1a59587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39649 } }
I20260811 12:38:40.801558 7724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:40.801772 7724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e81729cb13143e18cbd0f9ea1a59587, State: Initialized, Role: FOLLOWER
I20260811 12:38:40.803212 7724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e81729cb13143e18cbd0f9ea1a59587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39649 } }
I20260811 12:38:40.803648 7724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:40.803948 7724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:40.804349 7724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:40.809545 7724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e81729cb13143e18cbd0f9ea1a59587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39649 } }
I20260811 12:38:40.810683 7724 leader_election.cc:302] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e81729cb13143e18cbd0f9ea1a59587; no voters:
I20260811 12:38:40.811915 7724 leader_election.cc:288] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:40.812187 7740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:40.816285 7724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:40.820250 7740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [term 1 LEADER]: Becoming Leader. State: Replica: 8e81729cb13143e18cbd0f9ea1a59587, State: Running, Role: LEADER
I20260811 12:38:40.821998 7740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e81729cb13143e18cbd0f9ea1a59587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39649 } }
I20260811 12:38:40.827548 7742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e81729cb13143e18cbd0f9ea1a59587. Latest consensus state: current_term: 1 leader_uuid: "8e81729cb13143e18cbd0f9ea1a59587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e81729cb13143e18cbd0f9ea1a59587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39649 } } }
I20260811 12:38:40.828509 7742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:40.827633 7741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e81729cb13143e18cbd0f9ea1a59587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e81729cb13143e18cbd0f9ea1a59587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39649 } } }
I20260811 12:38:40.829445 7741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:40.840610 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:40.801837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48359 (local address 127.0.0.1:39649)
0811 12:38:40.802250 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:40.802260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:40.802968 (+ 708us) server_negotiation.cc:407] Connection header received
0811 12:38:40.803301 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:40.803314 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:40.803414 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:40.803557 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:40.807337 (+ 3780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:40.808764 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:40.811731 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:40.813251 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:40.815039 (+ 1788us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:40.819778 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:40.819798 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:40.819803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:40.819900 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:40.831319 (+ 11419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:40.833794 (+ 2475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:40.833811 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:40.833825 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:40.834244 (+ 419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:40.835489 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:40.835505 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:40.835510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:40.836453 (+ 943us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:40.838972 (+ 2519us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:40.839321 (+ 349us) server_negotiation.cc:299] Negotiation successful
0811 12:38:40.839718 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260811 12:38:40.843906 7754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:40.852777 7754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:40.869540 7754 catalog_manager.cc:1347] Generated new cluster ID: f9dc61cab5514f09ada1fb7d7ce561f3
I20260811 12:38:40.869743 7754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:40.891834 7754 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:40.894152 7754 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:40.916780 7754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e81729cb13143e18cbd0f9ea1a59587: Generated new TSK 0
I20260811 12:38:40.918394 7754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:40.929162 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7607
I20260811 12:38:40.931802 7716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:41.062419 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7607
W20260811 12:38:41.105233 7581 connection.cc:535] client connection to 127.0.0.1:39649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39649): Connection reset by peer (error 104)
2026-08-11T12:38:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2747 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:41 dist-test-slave-lgr5 krb5kdc[7774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:41 dist-test-slave-lgr5 krb5kdc[7774](info): set up 2 sockets
Aug 11 12:38:41 dist-test-slave-lgr5 krb5kdc[7774](info): commencing operation
krb5kdc: starting...
W20260811 12:38:42.458077 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.008s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:42 dist-test-slave-lgr5 krb5kdc[7774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:42.781416 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:43.134327 7797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:43.134660 7797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:43.134785 7797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:43.142591 7797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:43.142722 7797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:43.142800 7797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:43.142890 7797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:43.142954 7797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:43.153276 7797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:43.154742 7797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:43.156162 7797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:43.168356 7809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:43.168426 7805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:43.168411 7804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:43.169092 7797 server_base.cc:1034] running on GCE node
I20260811 12:38:43.170013 7797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:43.171538 7797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:43.172763 7797 hybrid_clock.cc:648] HybridClock initialized: now 1786451923172713 us; error 37 us; skew 500 ppm
I20260811 12:38:43.173168 7797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:43 dist-test-slave-lgr5 krb5kdc[7774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:43.179913 7797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:43.181582 7797 webserver.cc:460] Webserver started at http://127.0.0.1:33743/ using document root <none> and password file <none>
I20260811 12:38:43.182322 7797 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:43.182458 7797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:43.182763 7797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:43.185931 7797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "5999b7ceb4c44aca9d9b703f1a747bf7"
format_stamp: "Formatted at 2026-08-11 12:38:43 on dist-test-slave-lgr5"
server_key: "21a1e4f2c3ec001d7dd93ab16314e4b6"
server_key_iv: "40e0aec8bc34dd68705d5a09c45f0880"
server_key_version: "encryptionkey@0"
I20260811 12:38:43.186709 7797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "5999b7ceb4c44aca9d9b703f1a747bf7"
format_stamp: "Formatted at 2026-08-11 12:38:43 on dist-test-slave-lgr5"
server_key: "21a1e4f2c3ec001d7dd93ab16314e4b6"
server_key_iv: "40e0aec8bc34dd68705d5a09c45f0880"
server_key_version: "encryptionkey@0"
I20260811 12:38:43.192227 7797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260811 12:38:43.195804 7819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:43.197228 7797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:38:43.197436 7797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "5999b7ceb4c44aca9d9b703f1a747bf7"
format_stamp: "Formatted at 2026-08-11 12:38:43 on dist-test-slave-lgr5"
server_key: "21a1e4f2c3ec001d7dd93ab16314e4b6"
server_key_iv: "40e0aec8bc34dd68705d5a09c45f0880"
server_key_version: "encryptionkey@0"
I20260811 12:38:43.197646 7797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:43.217200 7797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:43.221738 7797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:43.222024 7797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:43.240557 7797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39351
I20260811 12:38:43.240592 7899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39351 every 8 connection(s)
I20260811 12:38:43.242005 7797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:43.244191 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7797
I20260811 12:38:43.244372 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:43.245031 3889 external_mini_cluster.cc:1428] Setting key 0b8bced8e9c62a3757f3109b493ece9c
I20260811 12:38:43.246574 7903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:43.258499 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:43.246831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56385 (local address 127.0.0.1:39351)
0811 12:38:43.247606 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:43.247617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:43.247657 (+ 40us) server_negotiation.cc:407] Connection header received
0811 12:38:43.249306 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:43.249328 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:43.249721 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:43.250066 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:43.250656 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:43.252014 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:43.253897 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:43.254591 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:43.254720 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:43.256069 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:43.256104 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:43.256123 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:43.256700 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:43.256786 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:43.256818 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:43.256961 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:43.257252 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 12:38:43.257447 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":333,"threads_started":1}
I20260811 12:38:43.259698 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7: Bootstrap starting.
I20260811 12:38:43.262418 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:43.263073 7903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:43.264364 7903 log.cc:824] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:43.267783 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7: No bootstrap required, opened a new log
I20260811 12:38:43.274612 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:43.267542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54135 (local address 127.0.0.1:39351)
0811 12:38:43.267719 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:43.267725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:43.268155 (+ 430us) server_negotiation.cc:407] Connection header received
0811 12:38:43.268360 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:43.268366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:43.268452 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:43.268573 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:43.269163 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:43.270314 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:43.271681 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:43.272413 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:43.272544 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:43.273741 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:43.273749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:43.273758 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:43.273781 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:43.273835 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:43.273862 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:43.273983 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:43.274209 (+ 226us) server_negotiation.cc:299] Negotiation successful
0811 12:38:43.274264 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 12:38:43.277194 7903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999b7ceb4c44aca9d9b703f1a747bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39351 } }
I20260811 12:38:43.277585 7903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:43.277686 7903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5999b7ceb4c44aca9d9b703f1a747bf7, State: Initialized, Role: FOLLOWER
I20260811 12:38:43.278280 7903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999b7ceb4c44aca9d9b703f1a747bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39351 } }
I20260811 12:38:43.278514 7903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:43.278644 7903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:43.278816 7903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:43.281180 7903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999b7ceb4c44aca9d9b703f1a747bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39351 } }
I20260811 12:38:43.281719 7903 leader_election.cc:302] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5999b7ceb4c44aca9d9b703f1a747bf7; no voters:
I20260811 12:38:43.282356 7903 leader_election.cc:288] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:43.282500 7928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:43.283211 7928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [term 1 LEADER]: Becoming Leader. State: Replica: 5999b7ceb4c44aca9d9b703f1a747bf7, State: Running, Role: LEADER
I20260811 12:38:43.283759 7928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999b7ceb4c44aca9d9b703f1a747bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39351 } }
I20260811 12:38:43.284610 7903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:43.286628 7929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5999b7ceb4c44aca9d9b703f1a747bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999b7ceb4c44aca9d9b703f1a747bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39351 } } }
I20260811 12:38:43.287235 7930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5999b7ceb4c44aca9d9b703f1a747bf7. Latest consensus state: current_term: 1 leader_uuid: "5999b7ceb4c44aca9d9b703f1a747bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999b7ceb4c44aca9d9b703f1a747bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39351 } } }
I20260811 12:38:43.287467 7929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:43.287561 7930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:43.289160 7939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:43.293802 7939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:43.302160 7939 catalog_manager.cc:1347] Generated new cluster ID: 87ad0110d7b9412bbdf84e93650cf3b7
I20260811 12:38:43.302301 7939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:43.315526 7939 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:43.317430 7939 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:43.328718 7939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5999b7ceb4c44aca9d9b703f1a747bf7: Generated new TSK 0
I20260811 12:38:43.329921 7939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:43.348884 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7797
W20260811 12:38:43.350692 7892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56385
W20260811 12:38:43.351480 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7797: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:38:43.351701 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7797
W20260811 12:38:43.377120 7772 connection.cc:535] client connection to 127.0.0.1:39351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39351): Connection reset by peer (error 104)
2026-08-11T12:38:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:43 dist-test-slave-lgr5 krb5kdc[7964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:43 dist-test-slave-lgr5 krb5kdc[7964](info): set up 2 sockets
Aug 11 12:38:43 dist-test-slave-lgr5 krb5kdc[7964](info): commencing operation
krb5kdc: starting...
W20260811 12:38:44.664202 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:44 dist-test-slave-lgr5 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:44.986989 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:45.341027 7980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:45.341339 7980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:45.341450 7980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:45.348760 7980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:45.348889 7980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:45.348943 7980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:45.349005 7980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:45.349048 7980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:45.359118 7980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:45.360441 7980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:45.361768 7980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:45.373684 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:45.373730 7991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:45.373762 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:45.374619 7980 server_base.cc:1034] running on GCE node
I20260811 12:38:45.375505 7980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:45.377058 7980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:45.378284 7980 hybrid_clock.cc:648] HybridClock initialized: now 1786451925378232 us; error 51 us; skew 500 ppm
I20260811 12:38:45.378697 7980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:45 dist-test-slave-lgr5 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:45.382215 7980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:45.383996 7980 webserver.cc:460] Webserver started at http://127.0.0.1:45429/ using document root <none> and password file <none>
I20260811 12:38:45.384757 7980 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:45.384883 7980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:45.385212 7980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:45.388296 7980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "028becfaf7b54433aaa20b032a4bf871"
format_stamp: "Formatted at 2026-08-11 12:38:45 on dist-test-slave-lgr5"
server_key: "d4bcc67342df3564826fa520a1d0db0c"
server_key_iv: "01db4875bd2cfae4d5d1cab4b5aba80d"
server_key_version: "encryptionkey@0"
I20260811 12:38:45.389078 7980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "028becfaf7b54433aaa20b032a4bf871"
format_stamp: "Formatted at 2026-08-11 12:38:45 on dist-test-slave-lgr5"
server_key: "d4bcc67342df3564826fa520a1d0db0c"
server_key_iv: "01db4875bd2cfae4d5d1cab4b5aba80d"
server_key_version: "encryptionkey@0"
I20260811 12:38:45.394380 7980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:38:45.397989 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:45.399526 7980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:38:45.399693 7980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "028becfaf7b54433aaa20b032a4bf871"
format_stamp: "Formatted at 2026-08-11 12:38:45 on dist-test-slave-lgr5"
server_key: "d4bcc67342df3564826fa520a1d0db0c"
server_key_iv: "01db4875bd2cfae4d5d1cab4b5aba80d"
server_key_version: "encryptionkey@0"
I20260811 12:38:45.399878 7980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:45.405807 7980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:45.409991 7980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:45.410251 7980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:45.428051 7980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40449
I20260811 12:38:45.428108 8087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40449 every 8 connection(s)
I20260811 12:38:45.429574 7980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:45.433532 8088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:45.439916 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 7980
I20260811 12:38:45.440161 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:45.440742 3889 external_mini_cluster.cc:1428] Setting key fe96ec5968f51f4ea8458f0a8bfaf126
I20260811 12:38:45.445073 8088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871: Bootstrap starting.
I20260811 12:38:45.448647 8088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:45.448798 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:45.442669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40975 (local address 127.0.0.1:40449)
0811 12:38:45.443684 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:45.443697 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:45.443740 (+ 43us) server_negotiation.cc:407] Connection header received
0811 12:38:45.445493 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:45.445518 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:45.445866 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:45.446188 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:45.447273 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:45.447280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:45.447292 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:45.447305 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:45.447384 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:45.447409 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:45.447567 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:45.447676 (+ 109us) server_negotiation.cc:299] Negotiation successful
0811 12:38:45.447845 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":413,"threads_started":1}
I20260811 12:38:45.449957 8088 log.cc:824] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:45.452733 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:45.453131 8088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871: No bootstrap required, opened a new log
I20260811 12:38:45.459602 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:45.457280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60111 (local address 127.0.0.1:40449)
0811 12:38:45.457466 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:45.457476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:45.457857 (+ 381us) server_negotiation.cc:407] Connection header received
0811 12:38:45.457992 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:45.457997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:45.458060 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:45.458169 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:45.458964 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:45.458970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:45.458975 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:45.458997 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:45.459044 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:45.459062 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:45.459184 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:45.459223 (+ 39us) server_negotiation.cc:299] Negotiation successful
0811 12:38:45.459276 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 12:38:45.461757 8088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028becfaf7b54433aaa20b032a4bf871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40449 } }
I20260811 12:38:45.462118 8088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:45.462196 8088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028becfaf7b54433aaa20b032a4bf871, State: Initialized, Role: FOLLOWER
I20260811 12:38:45.462704 8088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028becfaf7b54433aaa20b032a4bf871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40449 } }
I20260811 12:38:45.462927 8088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:45.463027 8088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:45.463157 8088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:45.465485 8088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028becfaf7b54433aaa20b032a4bf871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40449 } }
I20260811 12:38:45.465955 8088 leader_election.cc:302] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 028becfaf7b54433aaa20b032a4bf871; no voters:
I20260811 12:38:45.466482 8088 leader_election.cc:288] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:45.466616 8106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:45.467257 8106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [term 1 LEADER]: Becoming Leader. State: Replica: 028becfaf7b54433aaa20b032a4bf871, State: Running, Role: LEADER
I20260811 12:38:45.467821 8106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028becfaf7b54433aaa20b032a4bf871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40449 } }
I20260811 12:38:45.468600 8088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:45.470693 8108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "028becfaf7b54433aaa20b032a4bf871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028becfaf7b54433aaa20b032a4bf871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40449 } } }
I20260811 12:38:45.471086 8108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:45.471962 8110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 028becfaf7b54433aaa20b032a4bf871. Latest consensus state: current_term: 1 leader_uuid: "028becfaf7b54433aaa20b032a4bf871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028becfaf7b54433aaa20b032a4bf871" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40449 } } }
I20260811 12:38:45.472378 8110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:45.480655 8117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:45.486037 8117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:45.495533 8117 catalog_manager.cc:1347] Generated new cluster ID: bfba2fba12b14f6d9c0ace966d3e3892
I20260811 12:38:45.495668 8117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:45.508983 8117 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:45.509424 8117 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:45.516782 8117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 028becfaf7b54433aaa20b032a4bf871: Generated new TSK 0
I20260811 12:38:45.517843 8117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:45.533468 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7980
W20260811 12:38:45.534494 8081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40975
W20260811 12:38:45.535194 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 7980: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:38:45.535326 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 7980
2026-08-11T12:38:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2183 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:45 dist-test-slave-lgr5 krb5kdc[8146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:45 dist-test-slave-lgr5 krb5kdc[8146](info): set up 2 sockets
Aug 11 12:38:45 dist-test-slave-lgr5 krb5kdc[8146](info): commencing operation
krb5kdc: starting...
W20260811 12:38:46.818578 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.006s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:47 dist-test-slave-lgr5 krb5kdc[8146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:47.144611 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:47.502569 8163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:47.502905 8163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:47.503042 8163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:47.510399 8163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:47.510535 8163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:47.510627 8163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:47.510705 8163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:47.510764 8163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:47.520807 8163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:47.522140 8163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:47.523471 8163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:47.535190 8176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:47.535286 8178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:47.535259 8173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:47.536178 8163 server_base.cc:1034] running on GCE node
I20260811 12:38:47.537038 8163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:47.538508 8163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:47.539731 8163 hybrid_clock.cc:648] HybridClock initialized: now 1786451927539671 us; error 48 us; skew 500 ppm
I20260811 12:38:47.540068 8163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:47 dist-test-slave-lgr5 krb5kdc[8146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:47.543143 8163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:47.544725 8163 webserver.cc:460] Webserver started at http://127.0.0.1:40587/ using document root <none> and password file <none>
I20260811 12:38:47.545401 8163 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:47.545526 8163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:47.545823 8163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:47.548856 8163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "955124099d424e738f849c19c8847468"
format_stamp: "Formatted at 2026-08-11 12:38:47 on dist-test-slave-lgr5"
server_key: "274a900c3c59501675b9cfff2f3f59c6"
server_key_iv: "ac255d9168b49daff0afd77d5df0c39f"
server_key_version: "encryptionkey@0"
I20260811 12:38:47.549612 8163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "955124099d424e738f849c19c8847468"
format_stamp: "Formatted at 2026-08-11 12:38:47 on dist-test-slave-lgr5"
server_key: "274a900c3c59501675b9cfff2f3f59c6"
server_key_iv: "ac255d9168b49daff0afd77d5df0c39f"
server_key_version: "encryptionkey@0"
I20260811 12:38:47.554661 8163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:38:47.558163 8191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:47.559473 8163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:38:47.559657 8163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "955124099d424e738f849c19c8847468"
format_stamp: "Formatted at 2026-08-11 12:38:47 on dist-test-slave-lgr5"
server_key: "274a900c3c59501675b9cfff2f3f59c6"
server_key_iv: "ac255d9168b49daff0afd77d5df0c39f"
server_key_version: "encryptionkey@0"
I20260811 12:38:47.559845 8163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:47.572883 8163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:47.577219 8163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:47.577508 8163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:47.595533 8163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33641
I20260811 12:38:47.595552 8279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33641 every 8 connection(s)
I20260811 12:38:47.597047 8163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:47.600931 8281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:47.607360 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8163
I20260811 12:38:47.607573 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:47.608162 3889 external_mini_cluster.cc:1428] Setting key 0d60ba2616737a3c5f93e5d5051573ec
I20260811 12:38:47.612030 8281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468: Bootstrap starting.
I20260811 12:38:47.616034 8281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:47.617125 8281 log.cc:824] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:47.620230 8281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468: No bootstrap required, opened a new log
Aug 11 12:38:47 dist-test-slave-lgr5 krb5kdc[8146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 12:38:47.628608 8281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955124099d424e738f849c19c8847468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260811 12:38:47.629007 8281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:47.629112 8281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955124099d424e738f849c19c8847468, State: Initialized, Role: FOLLOWER
I20260811 12:38:47.629796 8281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955124099d424e738f849c19c8847468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260811 12:38:47.630103 8281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:47.630250 8281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:47.630466 8281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:47.633005 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:47.610026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58627 (local address 127.0.0.1:33641)
0811 12:38:47.610831 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:47.610844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:47.610887 (+ 43us) server_negotiation.cc:407] Connection header received
0811 12:38:47.612881 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:47.612913 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:47.613394 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:47.613742 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:47.615597 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:47.617118 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:47.618623 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:47.619319 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:47.619457 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:47.624537 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:47.624546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:47.624567 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:47.624630 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:47.628039 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:47.628826 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:47.628834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:47.628840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:47.629064 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:47.629547 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:47.629553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:47.629555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:47.630347 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:47.630667 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:47.631121 (+ 454us) server_negotiation.cc:299] Negotiation successful
0811 12:38:47.631602 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":369,"threads_started":1}
I20260811 12:38:47.633780 8281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955124099d424e738f849c19c8847468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260811 12:38:47.634369 8281 leader_election.cc:302] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 955124099d424e738f849c19c8847468; no voters:
I20260811 12:38:47.635030 8281 leader_election.cc:288] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:47.635197 8288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:47.635949 8288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [term 1 LEADER]: Becoming Leader. State: Replica: 955124099d424e738f849c19c8847468, State: Running, Role: LEADER
I20260811 12:38:47.636655 8288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955124099d424e738f849c19c8847468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260811 12:38:47.637277 8281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:47.637723 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:47.639433 8289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "955124099d424e738f849c19c8847468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955124099d424e738f849c19c8847468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } } }
I20260811 12:38:47.639906 8289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:47.646973 8291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955124099d424e738f849c19c8847468. Latest consensus state: current_term: 1 leader_uuid: "955124099d424e738f849c19c8847468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955124099d424e738f849c19c8847468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } } }
I20260811 12:38:47.647384 8295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:47.647418 8291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:47.653731 8295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:47.673216 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:47.650641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38871 (local address 127.0.0.1:33641)
0811 12:38:47.650818 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:47.650826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:47.652405 (+ 1579us) server_negotiation.cc:407] Connection header received
0811 12:38:47.652588 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:47.652594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:47.652681 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:47.652773 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:47.654984 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:47.656249 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:47.658208 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:47.658949 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:47.659056 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:47.662614 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:47.662621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:47.662628 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:47.662675 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:47.669298 (+ 6623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:47.670334 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:47.670340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:47.670345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:47.670527 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:47.671011 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:47.671017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:47.671021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:47.671521 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:47.671783 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:47.672517 (+ 734us) server_negotiation.cc:299] Negotiation successful
0811 12:38:47.672793 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 12:38:47.677273 8295 catalog_manager.cc:1347] Generated new cluster ID: f962f806aaa74098b564ae500e653d79
I20260811 12:38:47.677408 8295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:47.691536 8295 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:47.693537 8295 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:47.705960 8295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 955124099d424e738f849c19c8847468: Generated new TSK 0
I20260811 12:38:47.707232 8295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:47.713881 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8163
I20260811 12:38:47.715039 8273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:47.804314 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8163
2026-08-11T12:38:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:47 dist-test-slave-lgr5 krb5kdc[8329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:47 dist-test-slave-lgr5 krb5kdc[8329](info): set up 2 sockets
Aug 11 12:38:47 dist-test-slave-lgr5 krb5kdc[8329](info): commencing operation
krb5kdc: starting...
W20260811 12:38:49.085605 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.001s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:49 dist-test-slave-lgr5 krb5kdc[8329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:49.411598 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:49.758932 8358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:49.759241 8358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:49.759361 8358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:49.766793 8358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:49.766916 8358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:49.766973 8358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:49.767062 8358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:49.767102 8358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:49.777400 8358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:49.778782 8358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:49.780124 8358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:49.791981 8373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:49.792092 8365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:49.792114 8368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:49.792764 8358 server_base.cc:1034] running on GCE node
I20260811 12:38:49.793526 8358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:49.794945 8358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:49.796175 8358 hybrid_clock.cc:648] HybridClock initialized: now 1786451929796122 us; error 40 us; skew 500 ppm
I20260811 12:38:49.796525 8358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:49 dist-test-slave-lgr5 krb5kdc[8329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:49.802798 8358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:49.804508 8358 webserver.cc:460] Webserver started at http://127.0.0.1:34449/ using document root <none> and password file <none>
I20260811 12:38:49.805269 8358 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:49.805406 8358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:49.805724 8358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:49.809059 8358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "f9c200e102ce49b8a7e9f5863909f783"
format_stamp: "Formatted at 2026-08-11 12:38:49 on dist-test-slave-lgr5"
server_key: "b0e4c5824d9919202a31d23867612c56"
server_key_iv: "20aefee8a8977442788238ad0886e567"
server_key_version: "encryptionkey@0"
I20260811 12:38:49.809900 8358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "f9c200e102ce49b8a7e9f5863909f783"
format_stamp: "Formatted at 2026-08-11 12:38:49 on dist-test-slave-lgr5"
server_key: "b0e4c5824d9919202a31d23867612c56"
server_key_iv: "20aefee8a8977442788238ad0886e567"
server_key_version: "encryptionkey@0"
I20260811 12:38:49.815623 8358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260811 12:38:49.819446 8384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:49.820914 8358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 12:38:49.821105 8358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "f9c200e102ce49b8a7e9f5863909f783"
format_stamp: "Formatted at 2026-08-11 12:38:49 on dist-test-slave-lgr5"
server_key: "b0e4c5824d9919202a31d23867612c56"
server_key_iv: "20aefee8a8977442788238ad0886e567"
server_key_version: "encryptionkey@0"
I20260811 12:38:49.821305 8358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:49.841125 8358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:49.845603 8358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:49.845881 8358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:49.870090 8358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45437
I20260811 12:38:49.870127 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45437 every 8 connection(s)
I20260811 12:38:49.871516 8358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:49.875344 8486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:49.876904 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8358
I20260811 12:38:49.877092 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:49.877664 3889 external_mini_cluster.cc:1428] Setting key 9aceefa867b3330a001bf8124d4b067c
I20260811 12:38:49.889595 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783: Bootstrap starting.
I20260811 12:38:49.891250 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:49.879566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41723 (local address 127.0.0.1:45437)
0811 12:38:49.880448 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:49.880464 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:49.880508 (+ 44us) server_negotiation.cc:407] Connection header received
0811 12:38:49.882121 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:49.882145 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:49.882591 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:49.882921 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:49.883519 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:49.884879 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:49.886730 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:49.887502 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:49.887647 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:49.888925 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:49.888934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:49.888948 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:49.889510 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:49.889596 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:49.889623 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:49.889724 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:49.889997 (+ 273us) server_negotiation.cc:299] Negotiation successful
0811 12:38:49.890257 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":432,"threads_started":1}
I20260811 12:38:49.893227 8486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:49.894433 8486 log.cc:824] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:49.895242 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:49.897284 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783: No bootstrap required, opened a new log
I20260811 12:38:49.906297 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:49.899581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40231 (local address 127.0.0.1:45437)
0811 12:38:49.899780 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:49.899786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:49.900158 (+ 372us) server_negotiation.cc:407] Connection header received
0811 12:38:49.900370 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:49.900377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:49.900440 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:49.900526 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:49.901148 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:49.901908 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:49.903785 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:49.904356 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:49.904475 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:49.905624 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:49.905629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:49.905633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:49.905652 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:49.905693 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:49.905708 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:49.905773 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:49.905929 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 12:38:49.906006 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:38:49.907027 8486 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c200e102ce49b8a7e9f5863909f783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45437 } }
I20260811 12:38:49.907411 8486 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:49.907531 8486 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c200e102ce49b8a7e9f5863909f783, State: Initialized, Role: FOLLOWER
I20260811 12:38:49.908159 8486 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c200e102ce49b8a7e9f5863909f783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45437 } }
I20260811 12:38:49.908375 8486 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:49.908483 8486 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:49.908614 8486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:49.910835 8486 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c200e102ce49b8a7e9f5863909f783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45437 } }
I20260811 12:38:49.911319 8486 leader_election.cc:302] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9c200e102ce49b8a7e9f5863909f783; no voters:
I20260811 12:38:49.911840 8486 leader_election.cc:288] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:49.911971 8502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:49.912640 8502 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [term 1 LEADER]: Becoming Leader. State: Replica: f9c200e102ce49b8a7e9f5863909f783, State: Running, Role: LEADER
I20260811 12:38:49.913168 8502 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c200e102ce49b8a7e9f5863909f783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45437 } }
I20260811 12:38:49.913971 8486 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:49.916059 8507 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9c200e102ce49b8a7e9f5863909f783. Latest consensus state: current_term: 1 leader_uuid: "f9c200e102ce49b8a7e9f5863909f783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c200e102ce49b8a7e9f5863909f783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45437 } } }
I20260811 12:38:49.916049 8505 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9c200e102ce49b8a7e9f5863909f783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c200e102ce49b8a7e9f5863909f783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45437 } } }
I20260811 12:38:49.916507 8505 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:49.916507 8507 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:49.918076 8511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:49.923787 8511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:49.934542 8511 catalog_manager.cc:1347] Generated new cluster ID: dce1799abbd34381b9d30af813d5f88f
I20260811 12:38:49.934672 8511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:49.947816 8511 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:49.949353 8511 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:49.965847 8511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9c200e102ce49b8a7e9f5863909f783: Generated new TSK 0
I20260811 12:38:49.966802 8511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:49.978963 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8358
W20260811 12:38:49.981041 8476 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41723
W20260811 12:38:49.981948 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:38:49.982136 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8358
2026-08-11T12:38:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:50 dist-test-slave-lgr5 krb5kdc[8541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:50 dist-test-slave-lgr5 krb5kdc[8541](info): set up 2 sockets
Aug 11 12:38:50 dist-test-slave-lgr5 krb5kdc[8541](info): commencing operation
krb5kdc: starting...
W20260811 12:38:51.265756 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:51 dist-test-slave-lgr5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:51.593452 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:51.956314 8570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:51.956637 8570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:51.956779 8570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:51.964099 8570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:51.964227 8570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:51.964301 8570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:51.964391 8570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:51.964447 8570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:51.976646 8570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:51.978384 8570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:51.979732 8570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:51.991878 8580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:51.991852 8582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:51.992010 8579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:51.993055 8570 server_base.cc:1034] running on GCE node
I20260811 12:38:51.993990 8570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:51.995518 8570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:51.996735 8570 hybrid_clock.cc:648] HybridClock initialized: now 1786451931996685 us; error 40 us; skew 500 ppm
I20260811 12:38:51.997112 8570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:51 dist-test-slave-lgr5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:52.000268 8570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:52.001889 8570 webserver.cc:460] Webserver started at http://127.0.0.1:42841/ using document root <none> and password file <none>
I20260811 12:38:52.002632 8570 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:52.002758 8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:52.003062 8570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:52.006242 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "679c6287231d4e8984372ae34774cb30"
format_stamp: "Formatted at 2026-08-11 12:38:51 on dist-test-slave-lgr5"
server_key: "f5f137398dcc666789f9a2a8d27605d4"
server_key_iv: "8145b2ccad2260063194879172db9c2a"
server_key_version: "encryptionkey@0"
I20260811 12:38:52.007061 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "679c6287231d4e8984372ae34774cb30"
format_stamp: "Formatted at 2026-08-11 12:38:51 on dist-test-slave-lgr5"
server_key: "f5f137398dcc666789f9a2a8d27605d4"
server_key_iv: "8145b2ccad2260063194879172db9c2a"
server_key_version: "encryptionkey@0"
I20260811 12:38:52.013099 8570 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 12:38:52.016737 8596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:52.018059 8570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:52.018229 8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "679c6287231d4e8984372ae34774cb30"
format_stamp: "Formatted at 2026-08-11 12:38:51 on dist-test-slave-lgr5"
server_key: "f5f137398dcc666789f9a2a8d27605d4"
server_key_iv: "8145b2ccad2260063194879172db9c2a"
server_key_version: "encryptionkey@0"
I20260811 12:38:52.018416 8570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:52.024180 8570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:52.028204 8570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:52.028481 8570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:52.045621 8570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45765
I20260811 12:38:52.045667 8687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45765 every 8 connection(s)
I20260811 12:38:52.047155 8570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:52.051046 8689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:52.056784 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8570
I20260811 12:38:52.056951 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:52.057431 3889 external_mini_cluster.cc:1428] Setting key dfdb1d13a7e64c4da3d38882f85c2ffe
I20260811 12:38:52.063113 8689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30: Bootstrap starting.
I20260811 12:38:52.066779 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:52.059349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55089 (local address 127.0.0.1:45765)
0811 12:38:52.060292 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:52.060313 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:52.060368 (+ 55us) server_negotiation.cc:407] Connection header received
0811 12:38:52.062737 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:52.062769 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:52.063258 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:52.063740 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:52.064766 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:52.064776 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:52.064798 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:52.064817 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:52.064907 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:52.064953 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:52.065184 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:52.065322 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 12:38:52.065531 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":380,"threads_started":1}
I20260811 12:38:52.067305 8689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:52.068565 8689 log.cc:824] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:52.070921 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:52.071798 8689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30: No bootstrap required, opened a new log
I20260811 12:38:52.078161 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:52.075695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46639 (local address 127.0.0.1:45765)
0811 12:38:52.075893 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:52.075900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:52.076239 (+ 339us) server_negotiation.cc:407] Connection header received
0811 12:38:52.076389 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:52.076394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:52.076465 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:52.076585 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:52.077516 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:52.077522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:52.077526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:52.077549 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:52.077601 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:52.077624 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:52.077756 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:52.077797 (+ 41us) server_negotiation.cc:299] Negotiation successful
0811 12:38:52.077854 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 12:38:52.080235 8689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679c6287231d4e8984372ae34774cb30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45765 } }
I20260811 12:38:52.080612 8689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:52.080689 8689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679c6287231d4e8984372ae34774cb30, State: Initialized, Role: FOLLOWER
I20260811 12:38:52.081219 8689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679c6287231d4e8984372ae34774cb30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45765 } }
I20260811 12:38:52.081434 8689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:52.081534 8689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:52.081666 8689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:52.083841 8689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679c6287231d4e8984372ae34774cb30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45765 } }
I20260811 12:38:52.084385 8689 leader_election.cc:302] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 679c6287231d4e8984372ae34774cb30; no voters:
I20260811 12:38:52.084954 8689 leader_election.cc:288] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:52.085091 8707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:52.085734 8707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [term 1 LEADER]: Becoming Leader. State: Replica: 679c6287231d4e8984372ae34774cb30, State: Running, Role: LEADER
I20260811 12:38:52.086311 8707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679c6287231d4e8984372ae34774cb30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45765 } }
I20260811 12:38:52.087042 8689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:52.089313 8709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 679c6287231d4e8984372ae34774cb30. Latest consensus state: current_term: 1 leader_uuid: "679c6287231d4e8984372ae34774cb30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679c6287231d4e8984372ae34774cb30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45765 } } }
I20260811 12:38:52.089735 8709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:52.090029 8708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "679c6287231d4e8984372ae34774cb30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679c6287231d4e8984372ae34774cb30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45765 } } }
I20260811 12:38:52.090744 8708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:52.093070 8720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:52.098069 8720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:52.107470 8720 catalog_manager.cc:1347] Generated new cluster ID: 101adf11c9f141d18a5a434c173ea7a6
I20260811 12:38:52.107611 8720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:52.151750 8720 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:52.152244 8720 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:52.159507 8720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 679c6287231d4e8984372ae34774cb30: Generated new TSK 0
I20260811 12:38:52.160594 8720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:52.178670 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8570
W20260811 12:38:52.179847 8676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55089
W20260811 12:38:52.180634 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8570: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:38:52.180783 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8570
2026-08-11T12:38:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:52 dist-test-slave-lgr5 krb5kdc[8745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:52 dist-test-slave-lgr5 krb5kdc[8745](info): set up 2 sockets
Aug 11 12:38:52 dist-test-slave-lgr5 krb5kdc[8745](info): commencing operation
krb5kdc: starting...
W20260811 12:38:53.459014 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.002s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:53 dist-test-slave-lgr5 krb5kdc[8745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:53.787014 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:54.139101 8775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:54.139479 8775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:54.139627 8775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:54.147244 8775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:54.147404 8775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:54.147491 8775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:54.147585 8775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:54.147645 8775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:54.158092 8775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:54.159512 8775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:54.160902 8775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:54.172811 8783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:54.172859 8785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:54.172811 8791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:54.173794 8775 server_base.cc:1034] running on GCE node
I20260811 12:38:54.174679 8775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:54.176234 8775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:54.177489 8775 hybrid_clock.cc:648] HybridClock initialized: now 1786451934177415 us; error 66 us; skew 500 ppm
I20260811 12:38:54.177856 8775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:54 dist-test-slave-lgr5 krb5kdc[8745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:54.181078 8775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:54.182633 8775 webserver.cc:460] Webserver started at http://127.0.0.1:37493/ using document root <none> and password file <none>
I20260811 12:38:54.183341 8775 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:54.183442 8775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:54.183715 8775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:54.186960 8775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "068556fac05a4177a18b15e03eba6f8f"
format_stamp: "Formatted at 2026-08-11 12:38:54 on dist-test-slave-lgr5"
server_key: "eca3aefff6220c2dbd57530c6972414a"
server_key_iv: "2653fa90a62a9027746b622cd572f2c8"
server_key_version: "encryptionkey@0"
I20260811 12:38:54.187759 8775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "068556fac05a4177a18b15e03eba6f8f"
format_stamp: "Formatted at 2026-08-11 12:38:54 on dist-test-slave-lgr5"
server_key: "eca3aefff6220c2dbd57530c6972414a"
server_key_iv: "2653fa90a62a9027746b622cd572f2c8"
server_key_version: "encryptionkey@0"
I20260811 12:38:54.193259 8775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 12:38:54.196944 8808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:54.198379 8775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:54.198566 8775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "068556fac05a4177a18b15e03eba6f8f"
format_stamp: "Formatted at 2026-08-11 12:38:54 on dist-test-slave-lgr5"
server_key: "eca3aefff6220c2dbd57530c6972414a"
server_key_iv: "2653fa90a62a9027746b622cd572f2c8"
server_key_version: "encryptionkey@0"
I20260811 12:38:54.198753 8775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:54.259171 8775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:54.263523 8775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:54.263840 8775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:54.281774 8775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34053
I20260811 12:38:54.281810 8884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34053 every 8 connection(s)
I20260811 12:38:54.283172 8775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:54.287108 8888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:54.289973 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8775
I20260811 12:38:54.290146 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:54.290555 3889 external_mini_cluster.cc:1428] Setting key c68984d5dc082607977d792643586b60
I20260811 12:38:54.300066 8888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f: Bootstrap starting.
I20260811 12:38:54.303901 8888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:54.304963 8888 log.cc:824] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f: Log is configured to *not* fsync() on all Append() calls
Aug 11 12:38:54 dist-test-slave-lgr5 krb5kdc[8745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 12:38:54.308128 8888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f: No bootstrap required, opened a new log
I20260811 12:38:54.314921 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:54.292545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:46691 (local address 127.0.0.1:34053)
0811 12:38:54.293326 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:54.293342 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:54.293401 (+ 59us) server_negotiation.cc:407] Connection header received
0811 12:38:54.295780 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:54.295813 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:54.296430 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:54.296885 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:54.298235 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:54.300259 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:54.301580 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:54.302675 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:54.302864 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:54.307260 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:54.307268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:54.307284 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:54.307325 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:54.310451 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:54.311480 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:54.311486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:54.311493 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:54.311663 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:54.312190 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:54.312193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:54.312196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:54.312762 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:54.313027 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:54.313524 (+ 497us) server_negotiation.cc:299] Negotiation successful
0811 12:38:54.313847 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":313,"threads_started":1}
I20260811 12:38:54.316948 8888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068556fac05a4177a18b15e03eba6f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260811 12:38:54.317329 8888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:54.317457 8888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068556fac05a4177a18b15e03eba6f8f, State: Initialized, Role: FOLLOWER
I20260811 12:38:54.318081 8888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068556fac05a4177a18b15e03eba6f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260811 12:38:54.318326 8888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:54.318481 8888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:54.318673 8888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:54.318825 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:54.320984 8888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068556fac05a4177a18b15e03eba6f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260811 12:38:54.321502 8888 leader_election.cc:302] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 068556fac05a4177a18b15e03eba6f8f; no voters:
I20260811 12:38:54.322144 8888 leader_election.cc:288] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:54.322269 8909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:54.324661 8888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:54.324774 8909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [term 1 LEADER]: Becoming Leader. State: Replica: 068556fac05a4177a18b15e03eba6f8f, State: Running, Role: LEADER
I20260811 12:38:54.325491 8909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068556fac05a4177a18b15e03eba6f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260811 12:38:54.328653 8912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "068556fac05a4177a18b15e03eba6f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068556fac05a4177a18b15e03eba6f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } } }
I20260811 12:38:54.329088 8912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:54.329641 8913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 068556fac05a4177a18b15e03eba6f8f. Latest consensus state: current_term: 1 leader_uuid: "068556fac05a4177a18b15e03eba6f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068556fac05a4177a18b15e03eba6f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } } }
I20260811 12:38:54.329959 8913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:54.339969 8930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:54.341881 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:54.323463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:34081 (local address 127.0.0.1:34053)
0811 12:38:54.323666 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:54.323673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:54.324318 (+ 645us) server_negotiation.cc:407] Connection header received
0811 12:38:54.324496 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:54.324502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:54.324584 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:54.324690 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:54.326505 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:54.327339 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:54.329162 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:54.329930 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:54.330079 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:54.333772 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:54.333780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:54.333785 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:38:54.333834 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:54.337914 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:54.339073 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:54.339080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:54.339084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:54.339246 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:38:54.339836 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:38:54.339841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:38:54.339844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:38:54.340306 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:54.340602 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:54.341318 (+ 716us) server_negotiation.cc:299] Negotiation successful
0811 12:38:54.341527 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:38:54.345685 8930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:54.355841 8930 catalog_manager.cc:1347] Generated new cluster ID: fc59eb60cdb049189c2b68720d6b1890
I20260811 12:38:54.355980 8930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:54.367053 8930 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:54.368512 8930 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:54.382982 8930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 068556fac05a4177a18b15e03eba6f8f: Generated new TSK 0
I20260811 12:38:54.383895 8930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:54.399256 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8775
I20260811 12:38:54.400604 8876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:38:54.493649 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8775
W20260811 12:38:54.519038 8741 connection.cc:535] client connection to 127.0.0.1:34053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34053): Connection reset by peer (error 104)
2026-08-11T12:38:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:54 dist-test-slave-lgr5 krb5kdc[8954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:54 dist-test-slave-lgr5 krb5kdc[8954](info): set up 2 sockets
Aug 11 12:38:54 dist-test-slave-lgr5 krb5kdc[8954](info): commencing operation
krb5kdc: starting...
W20260811 12:38:55.773754 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:55 dist-test-slave-lgr5 krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:56.104475 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:56.455703 8983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:56.456007 8983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:56.456139 8983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:56.463480 8983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:56.463603 8983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:56.463665 8983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:56.463726 8983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:56.463770 8983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:56.473831 8983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:56.475199 8983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:56.476642 8983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:56.488543 8992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:56.488610 8995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:56.488564 8990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:56.489382 8983 server_base.cc:1034] running on GCE node
I20260811 12:38:56.490110 8983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:56.491494 8983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:56.492681 8983 hybrid_clock.cc:648] HybridClock initialized: now 1786451936492627 us; error 44 us; skew 500 ppm
I20260811 12:38:56.493005 8983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:56 dist-test-slave-lgr5 krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:56.504390 8983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:56.506057 8983 webserver.cc:460] Webserver started at http://127.0.0.1:42529/ using document root <none> and password file <none>
I20260811 12:38:56.506791 8983 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:56.506912 8983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:56.507227 8983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:56.510394 8983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "5826e176c3014b83a97a9cfa55d4b6b3"
format_stamp: "Formatted at 2026-08-11 12:38:56 on dist-test-slave-lgr5"
server_key: "2af7175e2b83cb5ec79270073b95cc4e"
server_key_iv: "5654bffc2211eb1982cbbf8d7dff90cc"
server_key_version: "encryptionkey@0"
I20260811 12:38:56.511221 8983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "5826e176c3014b83a97a9cfa55d4b6b3"
format_stamp: "Formatted at 2026-08-11 12:38:56 on dist-test-slave-lgr5"
server_key: "2af7175e2b83cb5ec79270073b95cc4e"
server_key_iv: "5654bffc2211eb1982cbbf8d7dff90cc"
server_key_version: "encryptionkey@0"
I20260811 12:38:56.516940 8983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 12:38:56.520561 9005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:56.521922 8983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 12:38:56.522084 8983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "5826e176c3014b83a97a9cfa55d4b6b3"
format_stamp: "Formatted at 2026-08-11 12:38:56 on dist-test-slave-lgr5"
server_key: "2af7175e2b83cb5ec79270073b95cc4e"
server_key_iv: "5654bffc2211eb1982cbbf8d7dff90cc"
server_key_version: "encryptionkey@0"
I20260811 12:38:56.522275 8983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:56.563194 8983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:56.567658 8983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:56.567938 8983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:56.585774 8983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39701
I20260811 12:38:56.585805 9095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39701 every 8 connection(s)
I20260811 12:38:56.587149 8983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:56.588153 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 8983
I20260811 12:38:56.588353 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:56.588883 3889 external_mini_cluster.cc:1428] Setting key 00dd3d7401a9e174edb85a2d11bfe664
I20260811 12:38:56.591984 9097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:56.601799 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:56.590808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:49067 (local address 127.0.0.1:39701)
0811 12:38:56.591592 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:56.591604 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:56.591643 (+ 39us) server_negotiation.cc:407] Connection header received
0811 12:38:56.593344 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:56.593371 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:56.593759 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:56.594113 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:56.594662 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:56.596005 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:56.597337 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:56.598032 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:56.598205 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:56.599443 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:56.599452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:56.599466 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:56.600055 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:56.600171 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:56.600202 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:56.600353 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:56.600639 (+ 286us) server_negotiation.cc:299] Negotiation successful
0811 12:38:56.600812 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":302,"threads_started":1}
I20260811 12:38:56.603101 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3: Bootstrap starting.
I20260811 12:38:56.605574 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:56.606402 9097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:56.607363 9097 log.cc:824] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:56.610287 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3: No bootstrap required, opened a new log
I20260811 12:38:56.615792 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:56.609821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:51195 (local address 127.0.0.1:39701)
0811 12:38:56.610005 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:56.610010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:56.610462 (+ 452us) server_negotiation.cc:407] Connection header received
0811 12:38:56.610644 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:56.610651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:56.610719 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:56.610802 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:56.611274 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:56.612063 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:56.613407 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:38:56.613864 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:38:56.613968 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:38:56.615092 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:56.615100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:56.615103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:56.615120 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:56.615162 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:56.615178 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:56.615282 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:56.615485 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 12:38:56.615541 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 12:38:56.618108 9097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826e176c3014b83a97a9cfa55d4b6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39701 } }
I20260811 12:38:56.618530 9097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:56.618624 9097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5826e176c3014b83a97a9cfa55d4b6b3, State: Initialized, Role: FOLLOWER
I20260811 12:38:56.619163 9097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826e176c3014b83a97a9cfa55d4b6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39701 } }
I20260811 12:38:56.619375 9097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:56.619508 9097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:56.619663 9097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:56.621867 9097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826e176c3014b83a97a9cfa55d4b6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39701 } }
I20260811 12:38:56.622365 9097 leader_election.cc:302] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5826e176c3014b83a97a9cfa55d4b6b3; no voters:
I20260811 12:38:56.623003 9097 leader_election.cc:288] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:56.623126 9110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:56.624130 9110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 5826e176c3014b83a97a9cfa55d4b6b3, State: Running, Role: LEADER
I20260811 12:38:56.624706 9110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826e176c3014b83a97a9cfa55d4b6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39701 } }
I20260811 12:38:56.625416 9097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:56.627898 9112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5826e176c3014b83a97a9cfa55d4b6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826e176c3014b83a97a9cfa55d4b6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39701 } } }
I20260811 12:38:56.628367 9112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:56.629959 9123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:56.627866 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5826e176c3014b83a97a9cfa55d4b6b3. Latest consensus state: current_term: 1 leader_uuid: "5826e176c3014b83a97a9cfa55d4b6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826e176c3014b83a97a9cfa55d4b6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39701 } } }
I20260811 12:38:56.630313 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3 [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:56.634477 9123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:56.643576 9123 catalog_manager.cc:1347] Generated new cluster ID: e82e3f21f161439b8ffdb12e60a5b948
I20260811 12:38:56.643800 9123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:56.657624 9123 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:56.659164 9123 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:56.665879 9123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5826e176c3014b83a97a9cfa55d4b6b3: Generated new TSK 0
I20260811 12:38:56.666656 9123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:56.674703 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8983
W20260811 12:38:56.676570 9090 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.158:49067
W20260811 12:38:56.677332 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 8983: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:38:56.677462 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 8983
W20260811 12:38:56.697467 8950 connection.cc:535] client connection to 127.0.0.1:39701 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39701): Connection reset by peer (error 104)
2026-08-11T12:38:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2177 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:56 dist-test-slave-lgr5 krb5kdc[9148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:56 dist-test-slave-lgr5 krb5kdc[9148](info): set up 2 sockets
Aug 11 12:38:56 dist-test-slave-lgr5 krb5kdc[9148](info): commencing operation
krb5kdc: starting...
W20260811 12:38:57.961225 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.005s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:38:58 dist-test-slave-lgr5 krb5kdc[9148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:38:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:38:58.291695 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:38:58.639138 9172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:38:58.639451 9172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:38:58.639586 9172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:38:58.646955 9172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:38:58.647083 9172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:38:58.647161 9172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:38:58.647246 9172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:38:58.647303 9172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:38:58.657629 9172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:38:58.659021 9172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:38:58.660399 9172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:38:58.672683 9184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:58.672711 9181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:38:58.672773 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:38:58.673357 9172 server_base.cc:1034] running on GCE node
I20260811 12:38:58.674233 9172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:38:58.675737 9172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:38:58.677000 9172 hybrid_clock.cc:648] HybridClock initialized: now 1786451938676935 us; error 53 us; skew 500 ppm
I20260811 12:38:58.677407 9172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:38:58 dist-test-slave-lgr5 krb5kdc[9148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:38:58.680615 9172 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:38:58.682154 9172 webserver.cc:460] Webserver started at http://127.0.0.1:38197/ using document root <none> and password file <none>
I20260811 12:38:58.682864 9172 fs_manager.cc:359] Metadata directory not provided
I20260811 12:38:58.682998 9172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:38:58.683302 9172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:38:58.686453 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "8b4e3d12e05847e8aa046bb46037569d"
format_stamp: "Formatted at 2026-08-11 12:38:58 on dist-test-slave-lgr5"
server_key: "e2c614f7b99fdc7d400b58e41ea3561b"
server_key_iv: "bcd0b3969829a9c151a6cfb9ea37c538"
server_key_version: "encryptionkey@0"
I20260811 12:38:58.687227 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "8b4e3d12e05847e8aa046bb46037569d"
format_stamp: "Formatted at 2026-08-11 12:38:58 on dist-test-slave-lgr5"
server_key: "e2c614f7b99fdc7d400b58e41ea3561b"
server_key_iv: "bcd0b3969829a9c151a6cfb9ea37c538"
server_key_version: "encryptionkey@0"
I20260811 12:38:58.692713 9172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 12:38:58.696741 9194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:38:58.698246 9172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:38:58.698426 9172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "8b4e3d12e05847e8aa046bb46037569d"
format_stamp: "Formatted at 2026-08-11 12:38:58 on dist-test-slave-lgr5"
server_key: "e2c614f7b99fdc7d400b58e41ea3561b"
server_key_iv: "bcd0b3969829a9c151a6cfb9ea37c538"
server_key_version: "encryptionkey@0"
I20260811 12:38:58.698633 9172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:38:58.704443 9172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:38:58.708803 9172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:38:58.709102 9172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:38:58.726778 9172 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46379
I20260811 12:38:58.726814 9292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46379 every 8 connection(s)
I20260811 12:38:58.728295 9172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:38:58.732265 9293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:38:58.733929 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9172
I20260811 12:38:58.734136 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:38:58.734717 3889 external_mini_cluster.cc:1428] Setting key c8ec3edd93b5f6576a2172ce34897c31
I20260811 12:38:58.742730 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:58.736791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:35563 (local address 127.0.0.1:46379)
0811 12:38:58.737654 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:58.737668 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:58.737719 (+ 51us) server_negotiation.cc:407] Connection header received
0811 12:38:58.739352 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:58.739383 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:58.739749 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:58.740159 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:58.741145 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:58.741151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:58.741164 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:58.741178 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:58.741243 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:58.741272 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:58.741496 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:58.741626 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 12:38:58.741814 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":371,"threads_started":1}
I20260811 12:38:58.746058 9293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d: Bootstrap starting.
I20260811 12:38:58.746486 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:38:58.749207 9293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d: Neither blocks nor log segments found. Creating new log.
I20260811 12:38:58.750339 9293 log.cc:824] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d: Log is configured to *not* fsync() on all Append() calls
I20260811 12:38:58.753494 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:38:58.750802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:40425 (local address 127.0.0.1:46379)
0811 12:38:58.751113 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0811 12:38:58.751121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:38:58.751547 (+ 426us) server_negotiation.cc:407] Connection header received
0811 12:38:58.751781 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:38:58.751786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:38:58.751865 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:38:58.752000 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:38:58.752881 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:38:58.752886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:38:58.752891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:38:58.752919 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:38:58.752969 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:38:58.752989 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:38:58.753110 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 12:38:58.753157 (+ 47us) server_negotiation.cc:299] Negotiation successful
0811 12:38:58.753220 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260811 12:38:58.753856 9293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d: No bootstrap required, opened a new log
I20260811 12:38:58.761456 9293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4e3d12e05847e8aa046bb46037569d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } }
I20260811 12:38:58.761886 9293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:38:58.762006 9293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4e3d12e05847e8aa046bb46037569d, State: Initialized, Role: FOLLOWER
I20260811 12:38:58.762650 9293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4e3d12e05847e8aa046bb46037569d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } }
I20260811 12:38:58.762881 9293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:38:58.763013 9293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:38:58.763159 9293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:38:58.765437 9293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4e3d12e05847e8aa046bb46037569d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } }
I20260811 12:38:58.765941 9293 leader_election.cc:302] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b4e3d12e05847e8aa046bb46037569d; no voters:
I20260811 12:38:58.766532 9293 leader_election.cc:288] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:38:58.766660 9312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:38:58.767302 9312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [term 1 LEADER]: Becoming Leader. State: Replica: 8b4e3d12e05847e8aa046bb46037569d, State: Running, Role: LEADER
I20260811 12:38:58.767818 9312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4e3d12e05847e8aa046bb46037569d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } }
I20260811 12:38:58.768584 9293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:38:58.770818 9315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b4e3d12e05847e8aa046bb46037569d. Latest consensus state: current_term: 1 leader_uuid: "8b4e3d12e05847e8aa046bb46037569d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4e3d12e05847e8aa046bb46037569d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } } }
I20260811 12:38:58.770977 9313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b4e3d12e05847e8aa046bb46037569d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4e3d12e05847e8aa046bb46037569d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } } }
I20260811 12:38:58.771282 9315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:58.771311 9313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d [sys.catalog]: This master's current role is: LEADER
I20260811 12:38:58.772303 9323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:38:58.777499 9323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:38:58.786587 9323 catalog_manager.cc:1347] Generated new cluster ID: 309afb5fbba3468bb5a01f2a5111fa68
I20260811 12:38:58.786720 9323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:38:58.818393 9323 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:38:58.818943 9323 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:38:58.836511 9323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b4e3d12e05847e8aa046bb46037569d: Generated new TSK 0
I20260811 12:38:58.837729 9323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:38:58.852648 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9172
W20260811 12:38:58.853850 9286 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.158:35563
W20260811 12:38:58.854594 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9172: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:38:58.854763 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9172
2026-08-11T12:38:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2182 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:38:58 dist-test-slave-lgr5 krb5kdc[9346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:38:58 dist-test-slave-lgr5 krb5kdc[9346](info): set up 2 sockets
Aug 11 12:38:58 dist-test-slave-lgr5 krb5kdc[9346](info): commencing operation
krb5kdc: starting...
W20260811 12:39:00.154075 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.002s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:00 dist-test-slave-lgr5 krb5kdc[9346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:39:00.484449 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:00.829543 9368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:00.829861 9368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:00.829977 9368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:00.837378 9368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:00.837502 9368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:00.837548 9368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:00.837610 9368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:00.837653 9368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:00.847592 9368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:00.848969 9368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:00.850265 9368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:00.861956 9378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:00.862028 9380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:00.862005 9385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:00.862938 9368 server_base.cc:1034] running on GCE node
I20260811 12:39:00.863762 9368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:00.865350 9368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:00.866566 9368 hybrid_clock.cc:648] HybridClock initialized: now 1786451940866518 us; error 47 us; skew 500 ppm
I20260811 12:39:00.866933 9368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:00 dist-test-slave-lgr5 krb5kdc[9346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:00.870077 9368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:39:00.871578 9368 webserver.cc:460] Webserver started at http://127.0.0.1:33679/ using document root <none> and password file <none>
I20260811 12:39:00.872363 9368 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:00.872485 9368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:00.872779 9368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:00.875881 9368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "759c0e8dc93d40e68e20c59ee52c490c"
format_stamp: "Formatted at 2026-08-11 12:39:00 on dist-test-slave-lgr5"
server_key: "c40f9ddf441236fa42aa9e3dae7e0208"
server_key_iv: "6c764f9b6010a5f5f39a403b81ff53ed"
server_key_version: "encryptionkey@0"
I20260811 12:39:00.876693 9368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "759c0e8dc93d40e68e20c59ee52c490c"
format_stamp: "Formatted at 2026-08-11 12:39:00 on dist-test-slave-lgr5"
server_key: "c40f9ddf441236fa42aa9e3dae7e0208"
server_key_iv: "6c764f9b6010a5f5f39a403b81ff53ed"
server_key_version: "encryptionkey@0"
I20260811 12:39:00.881968 9368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:39:00.885579 9395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:00.886965 9368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:39:00.887148 9368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "759c0e8dc93d40e68e20c59ee52c490c"
format_stamp: "Formatted at 2026-08-11 12:39:00 on dist-test-slave-lgr5"
server_key: "c40f9ddf441236fa42aa9e3dae7e0208"
server_key_iv: "6c764f9b6010a5f5f39a403b81ff53ed"
server_key_version: "encryptionkey@0"
I20260811 12:39:00.887347 9368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:00.917248 9368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:00.921553 9368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:00.921833 9368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:00.939929 9368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42183
I20260811 12:39:00.939956 9478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42183 every 8 connection(s)
I20260811 12:39:00.941382 9368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:00.945143 9482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:00.947167 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9368
I20260811 12:39:00.947350 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:00.947911 3889 external_mini_cluster.cc:1428] Setting key ee25b7f56e381cd06880b41784542822
I20260811 12:39:00.956815 9482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c: Bootstrap starting.
I20260811 12:39:00.960121 9482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:00.961193 9482 log.cc:824] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c: Log is configured to *not* fsync() on all Append() calls
Aug 11 12:39:00 dist-test-slave-lgr5 krb5kdc[9346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 12:39:00.964442 9482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c: No bootstrap required, opened a new log
I20260811 12:39:00.970487 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:00.949916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:60193 (local address 127.0.0.1:42183)
0811 12:39:00.950698 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:00.950713 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:00.950760 (+ 47us) server_negotiation.cc:407] Connection header received
0811 12:39:00.952530 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:00.952554 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:00.952956 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:00.953305 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:00.954764 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:00.956127 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:00.957471 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:00.958166 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:00.958349 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:00.962910 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:00.962918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:00.962944 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:00.962989 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:00.966010 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:00.967008 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:00.967015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:00.967025 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:00.967199 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:00.967671 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:00.967675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:00.967677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:00.968285 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:00.968548 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:00.969049 (+ 501us) server_negotiation.cc:299] Negotiation successful
0811 12:39:00.969362 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":315,"threads_started":1}
I20260811 12:39:00.973526 9482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c0e8dc93d40e68e20c59ee52c490c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260811 12:39:00.973904 9482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:00.974006 9482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759c0e8dc93d40e68e20c59ee52c490c, State: Initialized, Role: FOLLOWER
I20260811 12:39:00.974504 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:00.974646 9482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c0e8dc93d40e68e20c59ee52c490c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260811 12:39:00.974924 9482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:00.975073 9482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:00.975236 9482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:00.978519 9482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c0e8dc93d40e68e20c59ee52c490c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260811 12:39:00.979183 9482 leader_election.cc:302] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759c0e8dc93d40e68e20c59ee52c490c; no voters:
I20260811 12:39:00.979907 9482 leader_election.cc:288] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:00.980121 9496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:00.980758 9496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [term 1 LEADER]: Becoming Leader. State: Replica: 759c0e8dc93d40e68e20c59ee52c490c, State: Running, Role: LEADER
I20260811 12:39:00.981477 9496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c0e8dc93d40e68e20c59ee52c490c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260811 12:39:00.983562 9496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759c0e8dc93d40e68e20c59ee52c490c. Latest consensus state: current_term: 1 leader_uuid: "759c0e8dc93d40e68e20c59ee52c490c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c0e8dc93d40e68e20c59ee52c490c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } } }
I20260811 12:39:00.985179 9496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:00.984815 9503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759c0e8dc93d40e68e20c59ee52c490c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c0e8dc93d40e68e20c59ee52c490c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } } }
I20260811 12:39:00.985553 9503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:00.988477 9482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:00.988677 9505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:00.994253 9505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:01.005856 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:00.979893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:35429 (local address 127.0.0.1:42183)
0811 12:39:00.980133 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:00.980140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:00.986425 (+ 6285us) server_negotiation.cc:407] Connection header received
0811 12:39:00.986904 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:00.986908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:00.986975 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:00.987060 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:00.988611 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:00.989820 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:00.992067 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:00.992839 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:00.992961 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:00.996766 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:00.996777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:00.996781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:00.996826 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:01.001044 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:01.002627 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:01.002634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:01.002638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:01.002829 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:01.003509 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:01.003515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:01.003519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:01.004045 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:01.004394 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:01.005192 (+ 798us) server_negotiation.cc:299] Negotiation successful
0811 12:39:01.005427 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260811 12:39:01.015100 9505 catalog_manager.cc:1347] Generated new cluster ID: 7fae7c7b07eb407dbbf463ed3c09ed46
I20260811 12:39:01.015226 9505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:01.024302 9505 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:01.025677 9505 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:01.040238 9505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759c0e8dc93d40e68e20c59ee52c490c: Generated new TSK 0
I20260811 12:39:01.041231 9505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:01.053829 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9368
I20260811 12:39:01.055189 9469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:01.148607 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9368
W20260811 12:39:01.174322 9339 connection.cc:535] client connection to 127.0.0.1:42183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42183): Connection reset by peer (error 104)
2026-08-11T12:39:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:01 dist-test-slave-lgr5 krb5kdc[9531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:01 dist-test-slave-lgr5 krb5kdc[9531](info): set up 2 sockets
Aug 11 12:39:01 dist-test-slave-lgr5 krb5kdc[9531](info): commencing operation
krb5kdc: starting...
W20260811 12:39:02.445753 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:02 dist-test-slave-lgr5 krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:39:02.762995 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:03.113569 9551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:03.113916 9551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:03.114069 9551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:03.121606 9551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:03.121758 9551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:03.121841 9551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:03.121927 9551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:03.121995 9551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:03.132257 9551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37579
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:03.133704 9551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:03.135066 9551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:03.147046 9559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:03.147025 9562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:03.147053 9565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:03.147877 9551 server_base.cc:1034] running on GCE node
I20260811 12:39:03.148718 9551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:03.150149 9551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:03.151352 9551 hybrid_clock.cc:648] HybridClock initialized: now 1786451943151304 us; error 46 us; skew 500 ppm
I20260811 12:39:03.151695 9551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:03 dist-test-slave-lgr5 krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:03.155078 9551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:39:03.156826 9551 webserver.cc:460] Webserver started at http://127.0.0.1:41541/ using document root <none> and password file <none>
I20260811 12:39:03.157562 9551 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:03.157696 9551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:03.158031 9551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:03.161366 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "3a8a8ecc623a4066bc04d588890d6e80"
format_stamp: "Formatted at 2026-08-11 12:39:03 on dist-test-slave-lgr5"
server_key: "afc308aaf1a75715be94ed4794519bb1"
server_key_iv: "8b4743bb503b232169f509527ac4ebdb"
server_key_version: "encryptionkey@0"
I20260811 12:39:03.162186 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "3a8a8ecc623a4066bc04d588890d6e80"
format_stamp: "Formatted at 2026-08-11 12:39:03 on dist-test-slave-lgr5"
server_key: "afc308aaf1a75715be94ed4794519bb1"
server_key_iv: "8b4743bb503b232169f509527ac4ebdb"
server_key_version: "encryptionkey@0"
I20260811 12:39:03.167791 9551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:39:03.171717 9575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:03.173126 9551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:39:03.173305 9551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "3a8a8ecc623a4066bc04d588890d6e80"
format_stamp: "Formatted at 2026-08-11 12:39:03 on dist-test-slave-lgr5"
server_key: "afc308aaf1a75715be94ed4794519bb1"
server_key_iv: "8b4743bb503b232169f509527ac4ebdb"
server_key_version: "encryptionkey@0"
I20260811 12:39:03.173504 9551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:03.194873 9551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:03.199316 9551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:03.199612 9551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:03.217993 9551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37579
I20260811 12:39:03.218025 9672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37579 every 8 connection(s)
I20260811 12:39:03.219395 9551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:03.223268 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:03.225018 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9551
I20260811 12:39:03.225195 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:03.225767 3889 external_mini_cluster.cc:1428] Setting key 85e92280db8d7d3f94bec76dbe7bb19b
I20260811 12:39:03.237106 9674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80: Bootstrap starting.
I20260811 12:39:03.239283 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:03.227780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:53671 (local address 127.0.0.1:37579)
0811 12:39:03.228625 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:03.228640 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:03.228683 (+ 43us) server_negotiation.cc:407] Connection header received
0811 12:39:03.230293 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:03.230321 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:03.230774 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:03.231264 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:03.231930 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:03.233337 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:03.234668 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:03.235369 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:03.235550 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:03.236966 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:03.236974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:03.236990 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:03.237574 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:03.237658 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:03.237686 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:03.237833 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:03.238115 (+ 282us) server_negotiation.cc:299] Negotiation successful
0811 12:39:03.238294 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":361,"threads_started":1}
I20260811 12:39:03.240511 9674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:03.241566 9674 log.cc:824] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:03.243266 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:03.244951 9674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80: No bootstrap required, opened a new log
I20260811 12:39:03.255059 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:03.248476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:35337 (local address 127.0.0.1:37579)
0811 12:39:03.248658 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:03.248664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:03.248964 (+ 300us) server_negotiation.cc:407] Connection header received
0811 12:39:03.249125 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:03.249131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:03.249218 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:03.249330 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:03.249802 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:03.250955 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:03.252261 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:03.252974 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:03.253106 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:03.254272 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:03.254278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:03.254283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:03.254305 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:03.254376 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:03.254397 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:03.254513 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:03.254652 (+ 139us) server_negotiation.cc:299] Negotiation successful
0811 12:39:03.254711 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 12:39:03.254948 9674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8a8ecc623a4066bc04d588890d6e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37579 } }
I20260811 12:39:03.255481 9674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:03.255586 9674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8a8ecc623a4066bc04d588890d6e80, State: Initialized, Role: FOLLOWER
I20260811 12:39:03.256234 9674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8a8ecc623a4066bc04d588890d6e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37579 } }
I20260811 12:39:03.256494 9674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:03.256621 9674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:03.256780 9674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:03.258970 9674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8a8ecc623a4066bc04d588890d6e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37579 } }
I20260811 12:39:03.259509 9674 leader_election.cc:302] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a8a8ecc623a4066bc04d588890d6e80; no voters:
I20260811 12:39:03.260208 9674 leader_election.cc:288] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:03.260318 9689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:03.261013 9689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [term 1 LEADER]: Becoming Leader. State: Replica: 3a8a8ecc623a4066bc04d588890d6e80, State: Running, Role: LEADER
I20260811 12:39:03.262095 9689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8a8ecc623a4066bc04d588890d6e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37579 } }
I20260811 12:39:03.262873 9674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:03.264804 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a8a8ecc623a4066bc04d588890d6e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8a8ecc623a4066bc04d588890d6e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37579 } } }
I20260811 12:39:03.265301 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:03.265420 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a8a8ecc623a4066bc04d588890d6e80. Latest consensus state: current_term: 1 leader_uuid: "3a8a8ecc623a4066bc04d588890d6e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8a8ecc623a4066bc04d588890d6e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37579 } } }
I20260811 12:39:03.265791 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:03.267321 9701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:03.272855 9701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:03.283128 9701 catalog_manager.cc:1347] Generated new cluster ID: 7691b9544bd3415b9631d45846b79dc9
I20260811 12:39:03.283265 9701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:03.301025 9701 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:03.302366 9701 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:03.318194 9701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a8a8ecc623a4066bc04d588890d6e80: Generated new TSK 0
I20260811 12:39:03.319125 9701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:03.327566 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9551
W20260811 12:39:03.328855 9664 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.158:53671
W20260811 12:39:03.329741 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9551: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:39:03.329887 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9551
W20260811 12:39:03.358583 9529 connection.cc:535] client connection to 127.0.0.1:37579 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37579): Connection reset by peer (error 104)
2026-08-11T12:39:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2184 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:03 dist-test-slave-lgr5 krb5kdc[9719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:03 dist-test-slave-lgr5 krb5kdc[9719](info): set up 2 sockets
Aug 11 12:39:03 dist-test-slave-lgr5 krb5kdc[9719](info): commencing operation
krb5kdc: starting...
W20260811 12:39:04.603367 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:04 dist-test-slave-lgr5 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 12:39:04.923141 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:05.270499 9746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:05.270804 9746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:05.270907 9746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:05.278522 9746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:05.278661 9746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:05.278715 9746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:05.278779 9746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:05.278823 9746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:05.289143 9746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:05.290486 9746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:05.291828 9746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:05.304026 9756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:05.304104 9757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:05.304028 9760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:05.304983 9746 server_base.cc:1034] running on GCE node
I20260811 12:39:05.305796 9746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:05.307277 9746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:05.308504 9746 hybrid_clock.cc:648] HybridClock initialized: now 1786451945308455 us; error 40 us; skew 500 ppm
I20260811 12:39:05.308882 9746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:05 dist-test-slave-lgr5 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:05.312237 9746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 12:39:05.313848 9746 webserver.cc:460] Webserver started at http://127.0.0.1:39073/ using document root <none> and password file <none>
I20260811 12:39:05.314569 9746 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:05.314688 9746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:05.315019 9746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:05.318142 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "a37646ce12624690aa08b135308bc944"
format_stamp: "Formatted at 2026-08-11 12:39:05 on dist-test-slave-lgr5"
server_key: "a3bc09097a30b3e8386171fa632a9b57"
server_key_iv: "5bb01c3f65710991a32b741b84ab3cd4"
server_key_version: "encryptionkey@0"
I20260811 12:39:05.318948 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "a37646ce12624690aa08b135308bc944"
format_stamp: "Formatted at 2026-08-11 12:39:05 on dist-test-slave-lgr5"
server_key: "a3bc09097a30b3e8386171fa632a9b57"
server_key_iv: "5bb01c3f65710991a32b741b84ab3cd4"
server_key_version: "encryptionkey@0"
I20260811 12:39:05.324208 9746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:39:05.327726 9770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:05.329146 9746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:39:05.329310 9746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "a37646ce12624690aa08b135308bc944"
format_stamp: "Formatted at 2026-08-11 12:39:05 on dist-test-slave-lgr5"
server_key: "a3bc09097a30b3e8386171fa632a9b57"
server_key_iv: "5bb01c3f65710991a32b741b84ab3cd4"
server_key_version: "encryptionkey@0"
I20260811 12:39:05.329516 9746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:05.335513 9746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:05.339733 9746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:05.340024 9746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:05.358006 9746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38369
I20260811 12:39:05.358042 9864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38369 every 8 connection(s)
I20260811 12:39:05.359412 9746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:05.363332 9866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:05.365152 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9746
I20260811 12:39:05.365300 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:05.365732 3889 external_mini_cluster.cc:1428] Setting key 89962323501a99c2124b5bd04900b17d
I20260811 12:39:05.373970 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:05.367536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:33301 (local address 127.0.0.1:38369)
0811 12:39:05.368409 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:05.368425 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:05.368495 (+ 70us) server_negotiation.cc:407] Connection header received
0811 12:39:05.370147 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:05.370182 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:05.370674 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:05.371134 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:05.372062 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:05.372071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:05.372114 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:05.372132 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:05.372201 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:05.372233 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:05.372454 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:05.372577 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 12:39:05.372776 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":338,"threads_started":1}
I20260811 12:39:05.377705 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:05.377838 9866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944: Bootstrap starting.
I20260811 12:39:05.381928 9866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:05.383338 9866 log.cc:824] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:05.385718 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:05.383213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:50799 (local address 127.0.0.1:38369)
0811 12:39:05.383411 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:05.383419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:05.383658 (+ 239us) server_negotiation.cc:407] Connection header received
0811 12:39:05.383815 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:05.383821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:05.383887 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:05.384020 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:05.385087 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:05.385092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:05.385096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:05.385115 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:05.385162 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:05.385181 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:05.385304 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:05.385387 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 12:39:05.385445 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:39:05.387090 9866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944: No bootstrap required, opened a new log
I20260811 12:39:05.395534 9866 raft_consensus.cc:348] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37646ce12624690aa08b135308bc944" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } }
I20260811 12:39:05.395970 9866 raft_consensus.cc:374] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:05.396072 9866 raft_consensus.cc:729] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37646ce12624690aa08b135308bc944, State: Initialized, Role: FOLLOWER
I20260811 12:39:05.396736 9866 consensus_queue.cc:260] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37646ce12624690aa08b135308bc944" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } }
I20260811 12:39:05.396986 9866 raft_consensus.cc:388] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:05.397130 9866 raft_consensus.cc:482] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:05.397294 9866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:05.400135 9866 raft_consensus.cc:504] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37646ce12624690aa08b135308bc944" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } }
I20260811 12:39:05.400696 9866 leader_election.cc:302] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a37646ce12624690aa08b135308bc944; no voters:
I20260811 12:39:05.401335 9866 leader_election.cc:288] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:05.401492 9880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:05.402173 9880 raft_consensus.cc:686] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [term 1 LEADER]: Becoming Leader. State: Replica: a37646ce12624690aa08b135308bc944, State: Running, Role: LEADER
I20260811 12:39:05.402699 9880 consensus_queue.cc:237] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37646ce12624690aa08b135308bc944" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } }
I20260811 12:39:05.403524 9866 sys_catalog.cc:564] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:05.405761 9881 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a37646ce12624690aa08b135308bc944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37646ce12624690aa08b135308bc944" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } } }
I20260811 12:39:05.406183 9881 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:05.406023 9883 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a37646ce12624690aa08b135308bc944. Latest consensus state: current_term: 1 leader_uuid: "a37646ce12624690aa08b135308bc944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37646ce12624690aa08b135308bc944" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } } }
I20260811 12:39:05.406386 9883 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:05.409154 9896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:05.415442 9896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:05.424520 9896 catalog_manager.cc:1347] Generated new cluster ID: 346d877077c44d378fecbaeffafb01c0
I20260811 12:39:05.424640 9896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:05.469143 9896 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:05.469679 9896 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:05.476876 9896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a37646ce12624690aa08b135308bc944: Generated new TSK 0
I20260811 12:39:05.478020 9896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:05.489089 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9746
W20260811 12:39:05.490257 9856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.158:33301
W20260811 12:39:05.490988 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9746: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:39:05.491127 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9746
2026-08-11T12:39:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2157 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:05 dist-test-slave-lgr5 krb5kdc[9923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:05 dist-test-slave-lgr5 krb5kdc[9923](info): set up 2 sockets
Aug 11 12:39:05 dist-test-slave-lgr5 krb5kdc[9923](info): commencing operation
krb5kdc: starting...
W20260811 12:39:06.764668 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.004s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:06 dist-test-slave-lgr5 krb5kdc[9923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:39:07.089564 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46383
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:46383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:07.445276 9943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:07.445621 9943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:07.445761 9943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:07.453176 9943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:07.453313 9943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:07.453394 9943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:07.453475 9943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:07.453533 9943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:07.463630 9943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:46383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:07.465018 9943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:07.466361 9943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:07.478341 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:07.478399 9952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:07.478678 9951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:07.479598 9943 server_base.cc:1034] running on GCE node
I20260811 12:39:07.480463 9943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:07.481966 9943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:07.483160 9943 hybrid_clock.cc:648] HybridClock initialized: now 1786451947483109 us; error 49 us; skew 500 ppm
I20260811 12:39:07.483484 9943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:07 dist-test-slave-lgr5 krb5kdc[9923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:07.490651 9943 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:39:07.492345 9943 webserver.cc:460] Webserver started at http://127.3.204.126:41785/ using document root <none> and password file <none>
I20260811 12:39:07.493078 9943 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:07.493206 9943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:07.493521 9943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:07.496771 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "486045c711ab4466b25fa0ba3872d055"
format_stamp: "Formatted at 2026-08-11 12:39:07 on dist-test-slave-lgr5"
server_key: "a11954d831cafe36b59d366df7725e25"
server_key_iv: "5e8d425730bbd1f6c3f0b7a80981f1ec"
server_key_version: "encryptionkey@0"
I20260811 12:39:07.497596 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "486045c711ab4466b25fa0ba3872d055"
format_stamp: "Formatted at 2026-08-11 12:39:07 on dist-test-slave-lgr5"
server_key: "a11954d831cafe36b59d366df7725e25"
server_key_iv: "5e8d425730bbd1f6c3f0b7a80981f1ec"
server_key_version: "encryptionkey@0"
I20260811 12:39:07.502905 9943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:39:07.506588 9967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:07.507953 9943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:39:07.508149 9943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "486045c711ab4466b25fa0ba3872d055"
format_stamp: "Formatted at 2026-08-11 12:39:07 on dist-test-slave-lgr5"
server_key: "a11954d831cafe36b59d366df7725e25"
server_key_iv: "5e8d425730bbd1f6c3f0b7a80981f1ec"
server_key_version: "encryptionkey@0"
I20260811 12:39:07.508337 9943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:07.524325 9943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:07.528534 9943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:07.528806 9943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:07.546265 9943 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:46383
I20260811 12:39:07.546273 10047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:46383 every 8 connection(s)
I20260811 12:39:07.547609 9943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:07.551291 10049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:07.552692 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 9943
I20260811 12:39:07.552861 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:07.553303 3889 external_mini_cluster.cc:1428] Setting key 8b337ef21be0d41c9fb71c47dd58740f
I20260811 12:39:07.565783 10049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055: Bootstrap starting.
Aug 11 12:39:07 dist-test-slave-lgr5 krb5kdc[9923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:39:07.569036 10049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:07.570487 10049 log.cc:824] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:07.574175 10049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055: No bootstrap required, opened a new log
I20260811 12:39:07.577932 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:07.555169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46011 (local address 127.3.204.126:46383)
0811 12:39:07.556013 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:07.556034 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:07.556111 (+ 77us) server_negotiation.cc:407] Connection header received
0811 12:39:07.557895 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:07.557928 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:07.558496 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:07.558961 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:07.560394 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:07.562397 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:07.563748 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:07.564801 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:07.564986 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:07.569089 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:07.569100 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:07.569121 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:07.569177 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:07.572875 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:07.573822 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:07.573830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:07.573838 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:07.574058 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:07.574636 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:07.574641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:07.574645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:07.575424 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:07.575766 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:07.576144 (+ 378us) server_negotiation.cc:299] Negotiation successful
0811 12:39:07.576531 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":373,"threads_started":1}
I20260811 12:39:07.581947 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:07.583338 10049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486045c711ab4466b25fa0ba3872d055" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46383 } }
I20260811 12:39:07.583685 10049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:07.583789 10049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486045c711ab4466b25fa0ba3872d055, State: Initialized, Role: FOLLOWER
I20260811 12:39:07.584406 10049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486045c711ab4466b25fa0ba3872d055" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46383 } }
I20260811 12:39:07.584671 10049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:07.584810 10049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:07.584985 10049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:07.587750 10049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486045c711ab4466b25fa0ba3872d055" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46383 } }
I20260811 12:39:07.588469 10049 leader_election.cc:302] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486045c711ab4466b25fa0ba3872d055; no voters:
I20260811 12:39:07.589133 10049 leader_election.cc:288] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:07.589237 10066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:07.590833 10066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [term 1 LEADER]: Becoming Leader. State: Replica: 486045c711ab4466b25fa0ba3872d055, State: Running, Role: LEADER
I20260811 12:39:07.591409 10049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:07.591380 10066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486045c711ab4466b25fa0ba3872d055" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46383 } }
I20260811 12:39:07.595206 10068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486045c711ab4466b25fa0ba3872d055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486045c711ab4466b25fa0ba3872d055" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46383 } } }
I20260811 12:39:07.595433 10075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486045c711ab4466b25fa0ba3872d055. Latest consensus state: current_term: 1 leader_uuid: "486045c711ab4466b25fa0ba3872d055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486045c711ab4466b25fa0ba3872d055" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46383 } } }
I20260811 12:39:07.595647 10068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:07.595674 10075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:07.604851 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:07.586255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47551 (local address 127.3.204.126:46383)
0811 12:39:07.586457 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:07.586464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:07.586981 (+ 517us) server_negotiation.cc:407] Connection header received
0811 12:39:07.587139 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:07.587143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:07.587203 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:07.587286 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:07.589204 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:07.590465 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:07.592207 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:07.592973 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:07.593108 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:07.596435 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:07.596443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:07.596449 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:07.596505 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:07.600335 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:07.601433 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:07.601440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:07.601444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:07.601625 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:07.602220 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:07.602226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:07.602230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:07.602661 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:07.602939 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:07.604265 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0811 12:39:07.604485 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 12:39:07.609859 10085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:07.615537 10085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:07.624410 10085 catalog_manager.cc:1347] Generated new cluster ID: eebe14012fd344a29b551188683c0879
I20260811 12:39:07.624552 10085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:07.678562 10085 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:07.679987 10085 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:07.688686 10085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486045c711ab4466b25fa0ba3872d055: Generated new TSK 0
I20260811 12:39:07.689575 10085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:07.705016 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 9943
I20260811 12:39:07.706658 10043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:07.803238 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 9943
W20260811 12:39:07.827257 9916 connection.cc:535] client connection to 127.3.204.126:46383 recv error: Network error: failed to read from TLS socket (remote: 127.3.204.126:46383): Connection reset by peer (error 104)
2026-08-11T12:39:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:07 dist-test-slave-lgr5 krb5kdc[10108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:07 dist-test-slave-lgr5 krb5kdc[10108](info): set up 2 sockets
Aug 11 12:39:07 dist-test-slave-lgr5 krb5kdc[10108](info): commencing operation
krb5kdc: starting...
W20260811 12:39:09.078531 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:09 dist-test-slave-lgr5 krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:39:09.407521 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:36673
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:36673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:09.761323 10134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:09.761684 10134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:09.761837 10134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:09.769443 10134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:09.769580 10134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:09.769663 10134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:09.769758 10134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:09.769819 10134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:09.780447 10134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:36673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:36673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:09.782240 10134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:09.783774 10134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:09.796161 10139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:09.796065 10140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:09.796113 10142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:09.797116 10134 server_base.cc:1034] running on GCE node
I20260811 12:39:09.797896 10134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:09.799358 10134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:09.800567 10134 hybrid_clock.cc:648] HybridClock initialized: now 1786451949800510 us; error 49 us; skew 500 ppm
I20260811 12:39:09.800912 10134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:09 dist-test-slave-lgr5 krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451949, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:09.804316 10134 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:39:09.805984 10134 webserver.cc:460] Webserver started at http://127.3.204.126:35453/ using document root <none> and password file <none>
I20260811 12:39:09.806680 10134 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:09.806787 10134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:09.807075 10134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:09.810338 10134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "e6baf696d04b42e7aadd8ef99d50a2d2"
format_stamp: "Formatted at 2026-08-11 12:39:09 on dist-test-slave-lgr5"
server_key: "57eeece18a17683bb151e308e3ef157d"
server_key_iv: "5f3c9111c3e725acf4c622213fe7cf17"
server_key_version: "encryptionkey@0"
I20260811 12:39:09.811110 10134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "e6baf696d04b42e7aadd8ef99d50a2d2"
format_stamp: "Formatted at 2026-08-11 12:39:09 on dist-test-slave-lgr5"
server_key: "57eeece18a17683bb151e308e3ef157d"
server_key_iv: "5f3c9111c3e725acf4c622213fe7cf17"
server_key_version: "encryptionkey@0"
I20260811 12:39:09.816653 10134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:39:09.822039 10153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:09.824195 10134 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:39:09.824438 10134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "e6baf696d04b42e7aadd8ef99d50a2d2"
format_stamp: "Formatted at 2026-08-11 12:39:09 on dist-test-slave-lgr5"
server_key: "57eeece18a17683bb151e308e3ef157d"
server_key_iv: "5f3c9111c3e725acf4c622213fe7cf17"
server_key_version: "encryptionkey@0"
I20260811 12:39:09.824683 10134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:09.842460 10134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:09.847788 10134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:09.848104 10134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:09.868589 10134 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:36673
I20260811 12:39:09.868607 10232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:36673 every 8 connection(s)
I20260811 12:39:09.870157 10134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:09.871268 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10134
I20260811 12:39:09.871495 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:09.871901 3889 external_mini_cluster.cc:1428] Setting key 7dc4c6cba03d42119b7bc922c9c53f57
I20260811 12:39:09.875240 10233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:09.885728 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:09.873829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54799 (local address 127.3.204.126:36673)
0811 12:39:09.874633 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:09.874645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:09.874690 (+ 45us) server_negotiation.cc:407] Connection header received
0811 12:39:09.876591 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:09.876617 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:09.877020 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:09.877566 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:09.878197 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:09.879586 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:09.880971 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:09.881679 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:09.881872 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:09.883276 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:09.883284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:09.883299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:09.883901 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:09.883989 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:09.884021 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:09.884221 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:09.884500 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 12:39:09.884670 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":353,"threads_started":1}
I20260811 12:39:09.887925 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2: Bootstrap starting.
I20260811 12:39:09.890086 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:09.891538 10233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:09.892673 10233 log.cc:824] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:09.896107 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2: No bootstrap required, opened a new log
I20260811 12:39:09.901907 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:09.894485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33323 (local address 127.3.204.126:36673)
0811 12:39:09.894694 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:09.894700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:09.895101 (+ 401us) server_negotiation.cc:407] Connection header received
0811 12:39:09.895297 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:09.895302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:09.895367 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:09.895449 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:09.896205 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:09.897002 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:09.899014 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:09.899498 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:09.899607 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:09.901010 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:09.901015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:09.901018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:09.901037 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:09.901078 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:09.901094 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:09.901184 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:09.901464 (+ 280us) server_negotiation.cc:299] Negotiation successful
0811 12:39:09.901562 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 12:39:09.905845 10233 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6baf696d04b42e7aadd8ef99d50a2d2" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36673 } }
I20260811 12:39:09.906278 10233 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:09.906394 10233 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6baf696d04b42e7aadd8ef99d50a2d2, State: Initialized, Role: FOLLOWER
I20260811 12:39:09.907016 10233 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6baf696d04b42e7aadd8ef99d50a2d2" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36673 } }
I20260811 12:39:09.907266 10233 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:09.907413 10233 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:09.907594 10233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:09.910365 10233 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6baf696d04b42e7aadd8ef99d50a2d2" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36673 } }
I20260811 12:39:09.910912 10233 leader_election.cc:302] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6baf696d04b42e7aadd8ef99d50a2d2; no voters:
I20260811 12:39:09.911502 10233 leader_election.cc:288] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:09.911661 10257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:09.912470 10257 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [term 1 LEADER]: Becoming Leader. State: Replica: e6baf696d04b42e7aadd8ef99d50a2d2, State: Running, Role: LEADER
I20260811 12:39:09.913043 10257 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6baf696d04b42e7aadd8ef99d50a2d2" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36673 } }
I20260811 12:39:09.913846 10233 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:09.915935 10259 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6baf696d04b42e7aadd8ef99d50a2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6baf696d04b42e7aadd8ef99d50a2d2" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36673 } } }
I20260811 12:39:09.917241 10259 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:09.921435 10270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:09.917044 10260 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6baf696d04b42e7aadd8ef99d50a2d2. Latest consensus state: current_term: 1 leader_uuid: "e6baf696d04b42e7aadd8ef99d50a2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6baf696d04b42e7aadd8ef99d50a2d2" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 36673 } } }
I20260811 12:39:09.924505 10260 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:09.927057 10270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:09.939409 10270 catalog_manager.cc:1347] Generated new cluster ID: 7b4f28f864b94a4e966efa0798c625b4
I20260811 12:39:09.939535 10270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:09.957103 10270 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:09.958730 10270 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:09.970191 10270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6baf696d04b42e7aadd8ef99d50a2d2: Generated new TSK 0
I20260811 12:39:09.971333 10270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:09.986924 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10134
W20260811 12:39:09.989523 10228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54799
W20260811 12:39:09.990311 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10134: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:39:09.990451 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10134
W20260811 12:39:10.021997 10105 connection.cc:535] client connection to 127.3.204.126:36673 recv error: Network error: failed to read from TLS socket (remote: 127.3.204.126:36673): Connection reset by peer (error 104)
2026-08-11T12:39:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:10 dist-test-slave-lgr5 krb5kdc[10287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:10 dist-test-slave-lgr5 krb5kdc[10287](info): set up 2 sockets
Aug 11 12:39:10 dist-test-slave-lgr5 krb5kdc[10287](info): commencing operation
krb5kdc: starting...
W20260811 12:39:11.286616 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:11 dist-test-slave-lgr5 krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:39:11.609740 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37635
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:37635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:11.971683 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:11.971987 10314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:11.972213 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:11.981109 10314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:11.981244 10314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:11.981328 10314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:11.981411 10314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:11.981482 10314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:11.991614 10314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:37635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:37635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:11.993000 10314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:11.994297 10314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:12.008170 10321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:12.008208 10322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:12.008229 10324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:12.008776 10314 server_base.cc:1034] running on GCE node
I20260811 12:39:12.009577 10314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:12.011075 10314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:12.012285 10314 hybrid_clock.cc:648] HybridClock initialized: now 1786451952012230 us; error 44 us; skew 500 ppm
I20260811 12:39:12.012646 10314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:12 dist-test-slave-lgr5 krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451952, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:12.016374 10314 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:39:12.018096 10314 webserver.cc:460] Webserver started at http://127.3.204.126:38673/ using document root <none> and password file <none>
I20260811 12:39:12.018810 10314 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:12.018930 10314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:12.019230 10314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:12.022360 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "667a393b88a1456f8e4bb737feaa1813"
format_stamp: "Formatted at 2026-08-11 12:39:12 on dist-test-slave-lgr5"
server_key: "6259f241174ff9ff35d57053d7baa8db"
server_key_iv: "b62e8febe506db49c2d64e0ce51dab1d"
server_key_version: "encryptionkey@0"
I20260811 12:39:12.023164 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "667a393b88a1456f8e4bb737feaa1813"
format_stamp: "Formatted at 2026-08-11 12:39:12 on dist-test-slave-lgr5"
server_key: "6259f241174ff9ff35d57053d7baa8db"
server_key_iv: "b62e8febe506db49c2d64e0ce51dab1d"
server_key_version: "encryptionkey@0"
I20260811 12:39:12.028738 10314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:39:12.032290 10335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:12.033619 10314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:39:12.033790 10314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "667a393b88a1456f8e4bb737feaa1813"
format_stamp: "Formatted at 2026-08-11 12:39:12 on dist-test-slave-lgr5"
server_key: "6259f241174ff9ff35d57053d7baa8db"
server_key_iv: "b62e8febe506db49c2d64e0ce51dab1d"
server_key_version: "encryptionkey@0"
I20260811 12:39:12.033983 10314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:12.040122 10314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:12.044350 10314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:12.044641 10314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:12.062369 10314 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:37635
I20260811 12:39:12.062402 10407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:37635 every 8 connection(s)
I20260811 12:39:12.063814 10314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:12.067790 10408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:12.071203 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10314
I20260811 12:39:12.071393 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:12.071956 3889 external_mini_cluster.cc:1428] Setting key 4873d86b3d65d3d51fff5a79fd9082f1
I20260811 12:39:12.080502 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:12.074038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55663 (local address 127.3.204.126:37635)
0811 12:39:12.074871 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:12.074884 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:12.074928 (+ 44us) server_negotiation.cc:407] Connection header received
0811 12:39:12.076731 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:12.076758 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:12.077163 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:12.077577 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:12.078637 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:12.078644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:12.078659 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:12.078674 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:12.078741 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:12.078761 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:12.079080 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:12.079225 (+ 145us) server_negotiation.cc:299] Negotiation successful
0811 12:39:12.079459 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":373,"threads_started":1}
I20260811 12:39:12.080557 10408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813: Bootstrap starting.
I20260811 12:39:12.084146 10408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:12.084554 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:12.085263 10408 log.cc:824] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:12.088074 10408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813: No bootstrap required, opened a new log
I20260811 12:39:12.091687 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:12.089083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59343 (local address 127.3.204.126:37635)
0811 12:39:12.089306 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:12.089313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:12.089687 (+ 374us) server_negotiation.cc:407] Connection header received
0811 12:39:12.089864 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:12.089868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:12.089920 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:12.090015 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:12.090995 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:12.090999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:12.091005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:12.091022 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:12.091063 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:12.091079 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:12.091186 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:12.091330 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 12:39:12.091382 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:39:12.096505 10408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667a393b88a1456f8e4bb737feaa1813" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37635 } }
I20260811 12:39:12.096917 10408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:12.097012 10408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667a393b88a1456f8e4bb737feaa1813, State: Initialized, Role: FOLLOWER
I20260811 12:39:12.097587 10408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667a393b88a1456f8e4bb737feaa1813" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37635 } }
I20260811 12:39:12.097819 10408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:12.097946 10408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:12.098104 10408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:12.100387 10408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667a393b88a1456f8e4bb737feaa1813" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37635 } }
I20260811 12:39:12.100978 10408 leader_election.cc:302] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 667a393b88a1456f8e4bb737feaa1813; no voters:
I20260811 12:39:12.101866 10408 leader_election.cc:288] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:12.102042 10424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:12.102669 10424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [term 1 LEADER]: Becoming Leader. State: Replica: 667a393b88a1456f8e4bb737feaa1813, State: Running, Role: LEADER
I20260811 12:39:12.103329 10424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667a393b88a1456f8e4bb737feaa1813" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37635 } }
I20260811 12:39:12.104357 10408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:12.106343 10426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "667a393b88a1456f8e4bb737feaa1813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667a393b88a1456f8e4bb737feaa1813" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37635 } } }
I20260811 12:39:12.106518 10428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667a393b88a1456f8e4bb737feaa1813. Latest consensus state: current_term: 1 leader_uuid: "667a393b88a1456f8e4bb737feaa1813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667a393b88a1456f8e4bb737feaa1813" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 37635 } } }
I20260811 12:39:12.106868 10426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:12.106882 10428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:12.109333 10437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:12.114634 10437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:12.124725 10437 catalog_manager.cc:1347] Generated new cluster ID: 6d12597d69b549d9afffefc5c0165ab3
I20260811 12:39:12.124878 10437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:12.137717 10437 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:12.138262 10437 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:12.148844 10437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 667a393b88a1456f8e4bb737feaa1813: Generated new TSK 0
I20260811 12:39:12.149713 10437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:12.158118 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10314
W20260811 12:39:12.159348 10401 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55663
W20260811 12:39:12.160120 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10314: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:39:12.160285 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10314
2026-08-11T12:39:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2171 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:12 dist-test-slave-lgr5 krb5kdc[10462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:12 dist-test-slave-lgr5 krb5kdc[10462](info): set up 2 sockets
Aug 11 12:39:12 dist-test-slave-lgr5 krb5kdc[10462](info): commencing operation
krb5kdc: starting...
W20260811 12:39:13.444476 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:13 dist-test-slave-lgr5 krb5kdc[10462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:39:13.767400 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:34689
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:41387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:34689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:14.112777 10481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:14.113097 10481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:14.113224 10481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:14.120728 10481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:14.120859 10481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:14.120934 10481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:14.121026 10481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:14.121088 10481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:14.131232 10481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:41387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:34689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:14.132650 10481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:14.133992 10481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:14.146057 10496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:14.146231 10502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:14.146266 10490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:14.147326 10481 server_base.cc:1034] running on GCE node
I20260811 12:39:14.148188 10481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:14.149732 10481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:14.150951 10481 hybrid_clock.cc:648] HybridClock initialized: now 1786451954150897 us; error 40 us; skew 500 ppm
I20260811 12:39:14.151319 10481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:14 dist-test-slave-lgr5 krb5kdc[10462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451954, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:14.157979 10481 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:39:14.159653 10481 webserver.cc:460] Webserver started at http://127.3.204.126:34751/ using document root <none> and password file <none>
I20260811 12:39:14.160434 10481 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:14.160555 10481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:14.160866 10481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:14.164044 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "410f82833abd462a9cb04eae3ffb653f"
format_stamp: "Formatted at 2026-08-11 12:39:14 on dist-test-slave-lgr5"
server_key: "bd6628715bdc585391f83aebb286ec52"
server_key_iv: "1fa81513a57b87b8f5dcb66307184e6c"
server_key_version: "encryptionkey@0"
I20260811 12:39:14.164901 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "410f82833abd462a9cb04eae3ffb653f"
format_stamp: "Formatted at 2026-08-11 12:39:14 on dist-test-slave-lgr5"
server_key: "bd6628715bdc585391f83aebb286ec52"
server_key_iv: "1fa81513a57b87b8f5dcb66307184e6c"
server_key_version: "encryptionkey@0"
I20260811 12:39:14.170511 10481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 12:39:14.174250 10520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:14.175632 10481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:39:14.175803 10481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "410f82833abd462a9cb04eae3ffb653f"
format_stamp: "Formatted at 2026-08-11 12:39:14 on dist-test-slave-lgr5"
server_key: "bd6628715bdc585391f83aebb286ec52"
server_key_iv: "1fa81513a57b87b8f5dcb66307184e6c"
server_key_version: "encryptionkey@0"
I20260811 12:39:14.176007 10481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:14.192833 10481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:14.197294 10481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:14.197593 10481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:14.215616 10481 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:34689
I20260811 12:39:14.215660 10586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:34689 every 8 connection(s)
I20260811 12:39:14.217046 10481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:14.220191 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10481
I20260811 12:39:14.220359 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:14.220782 3889 external_mini_cluster.cc:1428] Setting key 974c025b71f67279bbd210c198acc678
I20260811 12:39:14.221026 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:14.235833 10587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f: Bootstrap starting.
Aug 11 12:39:14 dist-test-slave-lgr5 krb5kdc[10462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:39:14.240008 10587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:14.241323 10587 log.cc:824] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:14.245167 10587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f: No bootstrap required, opened a new log
I20260811 12:39:14.246142 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:14.222611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36627 (local address 127.3.204.126:34689)
0811 12:39:14.223411 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:14.223428 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:14.223479 (+ 51us) server_negotiation.cc:407] Connection header received
0811 12:39:14.225799 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:14.225831 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:14.226376 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:14.226844 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:14.228224 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:14.230196 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:14.231607 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:14.232846 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:14.233096 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:14.237754 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:14.237765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:14.237786 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:14.237836 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:14.241322 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:14.242429 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:14.242439 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:14.242448 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:14.242695 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:14.243245 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:14.243249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:14.243251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:14.243823 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:14.244106 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:14.244631 (+ 525us) server_negotiation.cc:299] Negotiation successful
0811 12:39:14.245030 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":327,"threads_started":1}
I20260811 12:39:14.250062 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:14.253698 10587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410f82833abd462a9cb04eae3ffb653f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34689 } }
I20260811 12:39:14.254122 10587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:14.254248 10587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410f82833abd462a9cb04eae3ffb653f, State: Initialized, Role: FOLLOWER
I20260811 12:39:14.255153 10587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410f82833abd462a9cb04eae3ffb653f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34689 } }
I20260811 12:39:14.255456 10587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:14.255611 10587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:14.255853 10587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:14.259451 10587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410f82833abd462a9cb04eae3ffb653f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34689 } }
I20260811 12:39:14.260028 10587 leader_election.cc:302] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 410f82833abd462a9cb04eae3ffb653f; no voters:
I20260811 12:39:14.260762 10587 leader_election.cc:288] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:14.260864 10607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:14.262207 10607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [term 1 LEADER]: Becoming Leader. State: Replica: 410f82833abd462a9cb04eae3ffb653f, State: Running, Role: LEADER
I20260811 12:39:14.262908 10607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410f82833abd462a9cb04eae3ffb653f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34689 } }
I20260811 12:39:14.263397 10587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:14.267071 10609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "410f82833abd462a9cb04eae3ffb653f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410f82833abd462a9cb04eae3ffb653f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34689 } } }
I20260811 12:39:14.267373 10609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:14.266361 10610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 410f82833abd462a9cb04eae3ffb653f. Latest consensus state: current_term: 1 leader_uuid: "410f82833abd462a9cb04eae3ffb653f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410f82833abd462a9cb04eae3ffb653f" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 34689 } } }
I20260811 12:39:14.267745 10610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:14.268415 10624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:14.274397 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:14.254597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36693 (local address 127.3.204.126:34689)
0811 12:39:14.254808 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:14.254814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:14.255224 (+ 410us) server_negotiation.cc:407] Connection header received
0811 12:39:14.255403 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:14.255408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:14.255502 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:14.255590 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:14.257755 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:14.258894 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:14.260251 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:14.261030 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:14.261208 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:14.264412 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:14.264419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:14.264425 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:14.264471 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:14.269028 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:14.270066 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:14.270073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:14.270077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:14.270260 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:14.270799 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:14.270805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:14.270808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:14.272952 (+ 2144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:14.273236 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:14.273716 (+ 480us) server_negotiation.cc:299] Negotiation successful
0811 12:39:14.273995 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 12:39:14.278976 10624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:14.290805 10624 catalog_manager.cc:1347] Generated new cluster ID: d671ed48e934429193b394533dce75eb
I20260811 12:39:14.290944 10624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:14.303711 10624 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:14.305199 10624 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:14.324173 10624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 410f82833abd462a9cb04eae3ffb653f: Generated new TSK 0
I20260811 12:39:14.325240 10624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:14.338771 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10481
I20260811 12:39:14.340123 10581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:14.433576 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10481
2026-08-11T12:39:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:14 dist-test-slave-lgr5 krb5kdc[10647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:14 dist-test-slave-lgr5 krb5kdc[10647](info): set up 2 sockets
Aug 11 12:39:14 dist-test-slave-lgr5 krb5kdc[10647](info): commencing operation
krb5kdc: starting...
W20260811 12:39:15.711162 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.005s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:15 dist-test-slave-lgr5 krb5kdc[10647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:39:16.039875 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33991
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:33991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:16.391680 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:16.391975 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:16.392124 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:16.399521 10670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:16.399667 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:16.399745 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:16.399839 10670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:16.399897 10670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:16.410074 10670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:33991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:33991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:16.411512 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:16.412942 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:16.425114 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:16.425179 10678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:16.425134 10677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:16.425832 10670 server_base.cc:1034] running on GCE node
I20260811 12:39:16.426689 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:16.428308 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:16.429553 10670 hybrid_clock.cc:648] HybridClock initialized: now 1786451956429505 us; error 42 us; skew 500 ppm
I20260811 12:39:16.429936 10670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:16 dist-test-slave-lgr5 krb5kdc[10647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:16.433156 10670 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:39:16.434629 10670 webserver.cc:460] Webserver started at http://127.3.204.126:35615/ using document root <none> and password file <none>
I20260811 12:39:16.435307 10670 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:16.435412 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:16.435675 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:16.438704 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "97a3ce7da58a4ec895bf4be0d7c95168"
format_stamp: "Formatted at 2026-08-11 12:39:16 on dist-test-slave-lgr5"
server_key: "434a4cd9cf84e24a3d56849eb60c5191"
server_key_iv: "f6347a57a9b49778969221c956f01f9b"
server_key_version: "encryptionkey@0"
I20260811 12:39:16.439445 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "97a3ce7da58a4ec895bf4be0d7c95168"
format_stamp: "Formatted at 2026-08-11 12:39:16 on dist-test-slave-lgr5"
server_key: "434a4cd9cf84e24a3d56849eb60c5191"
server_key_iv: "f6347a57a9b49778969221c956f01f9b"
server_key_version: "encryptionkey@0"
I20260811 12:39:16.444705 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 12:39:16.448278 10690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:16.449638 10670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 12:39:16.449808 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "97a3ce7da58a4ec895bf4be0d7c95168"
format_stamp: "Formatted at 2026-08-11 12:39:16 on dist-test-slave-lgr5"
server_key: "434a4cd9cf84e24a3d56849eb60c5191"
server_key_iv: "f6347a57a9b49778969221c956f01f9b"
server_key_version: "encryptionkey@0"
I20260811 12:39:16.449999 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:16.467278 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:16.471598 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:16.471873 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:16.489871 10670 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:33991
I20260811 12:39:16.489883 10772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:33991 every 8 connection(s)
I20260811 12:39:16.491313 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:16.491775 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10670
I20260811 12:39:16.491928 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:16.492527 3889 external_mini_cluster.cc:1428] Setting key 696066f3e5aec860177caeb49c267bbb
I20260811 12:39:16.495791 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:16.506479 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:16.494401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48957 (local address 127.3.204.126:33991)
0811 12:39:16.495353 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:16.495372 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:16.495959 (+ 587us) server_negotiation.cc:407] Connection header received
0811 12:39:16.497773 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:16.497796 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:16.498192 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:16.498576 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:16.499263 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:16.500697 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:16.502046 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:16.502716 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:16.502890 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:16.504176 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:16.504185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:16.504200 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:16.504789 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:16.504868 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:16.504894 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:16.505017 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:16.505243 (+ 226us) server_negotiation.cc:299] Negotiation successful
0811 12:39:16.505504 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":379,"threads_started":1}
I20260811 12:39:16.507176 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168: Bootstrap starting.
I20260811 12:39:16.510313 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:16.510986 10773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:16.512236 10773 log.cc:824] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:16.515561 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168: No bootstrap required, opened a new log
I20260811 12:39:16.521924 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:16.515358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43721 (local address 127.3.204.126:33991)
0811 12:39:16.515563 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:16.515568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:16.516050 (+ 482us) server_negotiation.cc:407] Connection header received
0811 12:39:16.516250 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:16.516255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:16.516335 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:16.516448 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:16.516927 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:16.518034 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:16.519324 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:16.520007 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:16.520167 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:16.521186 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:16.521193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:16.521198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:16.521220 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:16.521271 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:16.521290 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:16.521388 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:16.521460 (+ 72us) server_negotiation.cc:299] Negotiation successful
0811 12:39:16.521575 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 12:39:16.525107 10773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a3ce7da58a4ec895bf4be0d7c95168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33991 } }
I20260811 12:39:16.525489 10773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:16.525565 10773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a3ce7da58a4ec895bf4be0d7c95168, State: Initialized, Role: FOLLOWER
I20260811 12:39:16.526083 10773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a3ce7da58a4ec895bf4be0d7c95168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33991 } }
I20260811 12:39:16.526288 10773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:16.526381 10773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:16.526502 10773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:16.528769 10773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a3ce7da58a4ec895bf4be0d7c95168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33991 } }
I20260811 12:39:16.529387 10773 leader_election.cc:302] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97a3ce7da58a4ec895bf4be0d7c95168; no voters:
I20260811 12:39:16.530076 10773 leader_election.cc:288] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:16.530164 10788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:16.530810 10788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [term 1 LEADER]: Becoming Leader. State: Replica: 97a3ce7da58a4ec895bf4be0d7c95168, State: Running, Role: LEADER
I20260811 12:39:16.531432 10788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a3ce7da58a4ec895bf4be0d7c95168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33991 } }
I20260811 12:39:16.532199 10773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:16.534198 10791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97a3ce7da58a4ec895bf4be0d7c95168. Latest consensus state: current_term: 1 leader_uuid: "97a3ce7da58a4ec895bf4be0d7c95168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a3ce7da58a4ec895bf4be0d7c95168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33991 } } }
I20260811 12:39:16.534150 10790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97a3ce7da58a4ec895bf4be0d7c95168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a3ce7da58a4ec895bf4be0d7c95168" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 33991 } } }
I20260811 12:39:16.534600 10791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:16.534649 10790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:16.537499 10796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:16.542913 10796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:16.552428 10796 catalog_manager.cc:1347] Generated new cluster ID: abe4a0ffac804db68d038f65366f59bc
I20260811 12:39:16.552547 10796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:16.580713 10796 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:16.582103 10796 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:16.592147 10796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97a3ce7da58a4ec895bf4be0d7c95168: Generated new TSK 0
I20260811 12:39:16.592994 10796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:16.605208 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10670
W20260811 12:39:16.606369 10765 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48957
W20260811 12:39:16.607127 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:39:16.607262 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10670
2026-08-11T12:39:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2174 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:16 dist-test-slave-lgr5 krb5kdc[10828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:16 dist-test-slave-lgr5 krb5kdc[10828](info): set up 2 sockets
Aug 11 12:39:16 dist-test-slave-lgr5 krb5kdc[10828](info): commencing operation
krb5kdc: starting...
W20260811 12:39:17.894233 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:18 dist-test-slave-lgr5 krb5kdc[10828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:39:18.242791 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:44997
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:44997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:18.591133 10851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:18.591468 10851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:18.591614 10851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:18.598999 10851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:18.599150 10851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:18.599232 10851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:18.599313 10851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:18.599381 10851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:18.609678 10851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:44997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:44997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:18.611066 10851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:18.612406 10851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:18.623904 10865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:18.623977 10870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:18.624271 10863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:18.625259 10851 server_base.cc:1034] running on GCE node
I20260811 12:39:18.626041 10851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:18.627473 10851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:18.628674 10851 hybrid_clock.cc:648] HybridClock initialized: now 1786451958628624 us; error 40 us; skew 500 ppm
I20260811 12:39:18.629006 10851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:18 dist-test-slave-lgr5 krb5kdc[10828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451958, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:18.632048 10851 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:39:18.633590 10851 webserver.cc:460] Webserver started at http://127.3.204.126:37531/ using document root <none> and password file <none>
I20260811 12:39:18.634302 10851 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:18.634434 10851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:18.634722 10851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:18.637832 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "574a5aa1e3c3423b95c7e90bb47972b9"
format_stamp: "Formatted at 2026-08-11 12:39:18 on dist-test-slave-lgr5"
server_key: "874095c775493acbb1ecc52f309f3695"
server_key_iv: "b4e6558562c06f0265bb525ebcbd310f"
server_key_version: "encryptionkey@0"
I20260811 12:39:18.638615 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "574a5aa1e3c3423b95c7e90bb47972b9"
format_stamp: "Formatted at 2026-08-11 12:39:18 on dist-test-slave-lgr5"
server_key: "874095c775493acbb1ecc52f309f3695"
server_key_iv: "b4e6558562c06f0265bb525ebcbd310f"
server_key_version: "encryptionkey@0"
I20260811 12:39:18.643908 10851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260811 12:39:18.647411 10881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:18.648944 10851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:39:18.649115 10851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "574a5aa1e3c3423b95c7e90bb47972b9"
format_stamp: "Formatted at 2026-08-11 12:39:18 on dist-test-slave-lgr5"
server_key: "874095c775493acbb1ecc52f309f3695"
server_key_iv: "b4e6558562c06f0265bb525ebcbd310f"
server_key_version: "encryptionkey@0"
I20260811 12:39:18.649312 10851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:18.655486 10851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:18.659925 10851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:18.660288 10851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:18.678506 10851 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:44997
I20260811 12:39:18.678562 10969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:44997 every 8 connection(s)
I20260811 12:39:18.679970 10851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:18.684053 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:18.684142 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 10851
I20260811 12:39:18.684305 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:18.684851 3889 external_mini_cluster.cc:1428] Setting key ad6abfed5f6310e19bc6ef051ab51cbf
I20260811 12:39:18.692845 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:18.686781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58751 (local address 127.3.204.126:44997)
0811 12:39:18.687730 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:18.687742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:18.687790 (+ 48us) server_negotiation.cc:407] Connection header received
0811 12:39:18.689408 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:18.689434 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:18.689796 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:18.690197 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:18.691242 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:18.691249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:18.691262 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:18.691277 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:18.691362 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:18.691383 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:18.691600 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:18.691733 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 12:39:18.691913 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":454,"threads_started":1}
I20260811 12:39:18.696460 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:18.697588 10971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9: Bootstrap starting.
I20260811 12:39:18.700811 10971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:18.702071 10971 log.cc:824] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:18.703224 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:18.700759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44629 (local address 127.3.204.126:44997)
0811 12:39:18.700946 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:18.700952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:18.701326 (+ 374us) server_negotiation.cc:407] Connection header received
0811 12:39:18.701469 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:18.701475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:18.701537 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:18.701645 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:18.702559 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:18.702567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:18.702576 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 12:39:18.702597 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:18.702651 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 12:39:18.702670 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:18.702800 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:18.702910 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 12:39:18.702959 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:39:18.705235 10971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9: No bootstrap required, opened a new log
I20260811 12:39:18.712970 10971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574a5aa1e3c3423b95c7e90bb47972b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44997 } }
I20260811 12:39:18.713353 10971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 12:39:18.713454 10971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574a5aa1e3c3423b95c7e90bb47972b9, State: Initialized, Role: FOLLOWER
I20260811 12:39:18.714020 10971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574a5aa1e3c3423b95c7e90bb47972b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44997 } }
I20260811 12:39:18.714243 10971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 12:39:18.714382 10971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 12:39:18.714524 10971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:18.716835 10971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574a5aa1e3c3423b95c7e90bb47972b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44997 } }
I20260811 12:39:18.717340 10971 leader_election.cc:302] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 574a5aa1e3c3423b95c7e90bb47972b9; no voters:
I20260811 12:39:18.717917 10971 leader_election.cc:288] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 12:39:18.718062 10988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:18.718714 10988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [term 1 LEADER]: Becoming Leader. State: Replica: 574a5aa1e3c3423b95c7e90bb47972b9, State: Running, Role: LEADER
I20260811 12:39:18.719259 10988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574a5aa1e3c3423b95c7e90bb47972b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44997 } }
I20260811 12:39:18.719997 10971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:18.722398 10990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574a5aa1e3c3423b95c7e90bb47972b9. Latest consensus state: current_term: 1 leader_uuid: "574a5aa1e3c3423b95c7e90bb47972b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574a5aa1e3c3423b95c7e90bb47972b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44997 } } }
I20260811 12:39:18.722865 10990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:18.722962 10989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "574a5aa1e3c3423b95c7e90bb47972b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574a5aa1e3c3423b95c7e90bb47972b9" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 44997 } } }
I20260811 12:39:18.723248 10989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:18.724629 10998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:18.729743 10998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:18.739903 10998 catalog_manager.cc:1347] Generated new cluster ID: 1b204351c693483fad2e5a46fcf59985
I20260811 12:39:18.740025 10998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:18.759152 10998 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:18.759604 10998 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:18.769027 10998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 574a5aa1e3c3423b95c7e90bb47972b9: Generated new TSK 0
I20260811 12:39:18.769979 10998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:18.784549 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10851
W20260811 12:39:18.785732 10959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58751
W20260811 12:39:18.786476 3889 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 10851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 12:39:18.786617 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 10851
2026-08-11T12:39:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2179 ms)
[----------] 18 tests from AuthTokenIssuingTest (40449 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:18 dist-test-slave-lgr5 krb5kdc[11021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:18 dist-test-slave-lgr5 krb5kdc[11021](info): set up 2 sockets
Aug 11 12:39:18 dist-test-slave-lgr5 krb5kdc[11021](info): commencing operation
krb5kdc: starting...
W20260811 12:39:20.069391 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.006s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:20 dist-test-slave-lgr5 krb5kdc[11021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:39:20.400422 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:40599
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:40599,127.3.204.125:38733,127.3.204.124:35029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:20.749852 11044 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 12:39:20.750187 11044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:20.750268 11044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:20.750352 11044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:20.757695 11044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:20.757802 11044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:20.757860 11044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:20.757911 11044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:20.757974 11044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:20.767989 11044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:40599,127.3.204.125:38733,127.3.204.124:35029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:40599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:20.769368 11044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:20.770694 11044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:20.782475 11050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:20.782531 11051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:20.782663 11053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:20.783330 11044 server_base.cc:1034] running on GCE node
I20260811 12:39:20.784132 11044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:20.785540 11044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:20.786762 11044 hybrid_clock.cc:648] HybridClock initialized: now 1786451960786695 us; error 41 us; skew 500 ppm
I20260811 12:39:20.787099 11044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:20 dist-test-slave-lgr5 krb5kdc[11021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:20.794464 11044 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:39:20.796201 11044 webserver.cc:460] Webserver started at http://127.3.204.126:37253/ using document root <none> and password file <none>
I20260811 12:39:20.796941 11044 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:20.797050 11044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:20.797381 11044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:20.800580 11044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "d6a57e25614543dba255a41aff47d1fa"
format_stamp: "Formatted at 2026-08-11 12:39:20 on dist-test-slave-lgr5"
server_key: "14dfeedfd816a7f270db7095e41d02d3"
server_key_iv: "d76f1be2b7e74de9d2badce3bfe83bd7"
server_key_version: "encryptionkey@0"
I20260811 12:39:20.801380 11044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "d6a57e25614543dba255a41aff47d1fa"
format_stamp: "Formatted at 2026-08-11 12:39:20 on dist-test-slave-lgr5"
server_key: "14dfeedfd816a7f270db7095e41d02d3"
server_key_iv: "d76f1be2b7e74de9d2badce3bfe83bd7"
server_key_version: "encryptionkey@0"
I20260811 12:39:20.806905 11044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:39:20.810446 11067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:20.811797 11044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 12:39:20.811972 11044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "d6a57e25614543dba255a41aff47d1fa"
format_stamp: "Formatted at 2026-08-11 12:39:20 on dist-test-slave-lgr5"
server_key: "14dfeedfd816a7f270db7095e41d02d3"
server_key_iv: "d76f1be2b7e74de9d2badce3bfe83bd7"
server_key_version: "encryptionkey@0"
I20260811 12:39:20.812237 11044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:20.839107 11044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:20.843268 11044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:20.843544 11044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:20.861236 11044 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:40599
I20260811 12:39:20.861243 11150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:40599 every 8 connection(s)
I20260811 12:39:20.862653 11044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:20.863124 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11044
I20260811 12:39:20.863330 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:20.864158 3889 external_mini_cluster.cc:1428] Setting key 3ef5c4f5f23c8dd85af15abfce3728f9
I20260811 12:39:20.866690 11151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:20.873766 11151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:20 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.204.125@KRBTEST.COM; defaulting to no policy
I20260811 12:39:20.897300 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:20.878005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.3.204.126:40599)
0811 12:39:20.878597 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:20.878603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:20.878643 (+ 40us) server_negotiation.cc:407] Connection header received
0811 12:39:20.880617 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:20.880623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:20.880841 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:20.880999 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:20.882650 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:20.883830 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:20.885783 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:20.886339 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:20.886496 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:20.890424 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:20.890433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:20.890447 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:20.890479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:20.893130 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:20.893972 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:20.893979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:20.893981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:20.894128 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:20.894573 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:20.894576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:20.894578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:20.895082 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:20.895241 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:20.895921 (+ 680us) server_negotiation.cc:299] Negotiation successful
0811 12:39:20.896241 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":425,"thread_start_us":276,"threads_started":1}
I20260811 12:39:20.897370 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:20.876933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40599 (local address 127.0.0.1:48936)
0811 12:39:20.877773 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:20.877827 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:20.880242 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:20.881092 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:20.881099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:20.881889 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:20.882420 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:20.882445 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:20.883997 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:20.884002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:20.885629 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:20.885649 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:20.886595 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:20.886603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:20.886779 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:20.887474 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:20.887525 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:20.890171 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:20.893258 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:20.893272 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:20.893285 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:20.893836 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:20.894239 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:20.894244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:20.894248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:20.894465 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:20.895227 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:20.895235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:20.895482 (+ 247us) client_negotiation.cc:769] Sending connection context
0811 12:39:20.895774 (+ 292us) client_negotiation.cc:240] Negotiation successful
0811 12:39:20.896418 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":119,"thread_start_us":311,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.125@KRBTEST.COM" created.
I20260811 12:39:20.900420 11151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:39:20.901700 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:20.901227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40378)
0811 12:39:20.901416 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:20.901548 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260811 12:39:20.902041 11070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
I20260811 12:39:20.905380 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:20.904954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40388)
0811 12:39:20.905115 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:20.905201 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 12:39:20.905668 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:38733: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab.
Entry for principal kudu/127.3.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab.
I20260811 12:39:20.937237 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.125:38733
--webserver_interface=127.3.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab
--principal=kudu/127.3.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:40599,127.3.204.125:38733,127.3.204.124:35029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:39:20.955070 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } attempt: 1
I20260811 12:39:20.956336 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:20.955854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40390)
0811 12:39:20.956054 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:20.956201 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 12:39:20.957267 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:20.956929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40394)
0811 12:39:20.957071 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:20.957138 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 12:39:20.957444 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:38733: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
I20260811 12:39:21.025694 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } attempt: 2
I20260811 12:39:21.026873 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.026405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40400)
0811 12:39:21.026639 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.026731 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260811 12:39:21.027828 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.027463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40416)
0811 12:39:21.027623 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.027695 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 12:39:21.028012 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:38733: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
I20260811 12:39:21.119277 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } attempt: 3
I20260811 12:39:21.120402 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.119936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40424)
0811 12:39:21.120175 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.120267 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260811 12:39:21.121577 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.121200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40426)
0811 12:39:21.121343 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.121420 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 12:39:21.121788 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:38733: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
I20260811 12:39:21.265082 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } attempt: 4
I20260811 12:39:21.266374 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.265803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40438)
0811 12:39:21.266150 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.266239 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260811 12:39:21.267354 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.267010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40450)
0811 12:39:21.267153 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.267235 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 12:39:21.267537 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:38733: Network error: Client connection negotiation failed: client connection to 127.3.204.125:38733: connect: Connection refused (error 111)
W20260811 12:39:21.290438 11167 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 12:39:21.290817 11167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:21.290917 11167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:21.291031 11167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:21.298624 11167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:21.298730 11167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:21.298811 11167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:21.298877 11167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:21.298967 11167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:21.309104 11167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:40599,127.3.204.125:38733,127.3.204.124:35029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab
--principal=kudu/127.3.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.125:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:21.310544 11167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:21.311902 11167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:21.323601 11175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:21.323623 11178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:21.324323 11180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:21.324518 11167 server_base.cc:1034] running on GCE node
I20260811 12:39:21.325505 11167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:21.327023 11167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:21.328251 11167 hybrid_clock.cc:648] HybridClock initialized: now 1786451961328194 us; error 30 us; skew 500 ppm
I20260811 12:39:21.328588 11167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:21 dist-test-slave-lgr5 krb5kdc[11021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:21.331794 11167 init.cc:377] Logged in from keytab as kudu/127.3.204.125@KRBTEST.COM (short username kudu)
I20260811 12:39:21.333282 11167 webserver.cc:460] Webserver started at http://127.3.204.125:44203/ using document root <none> and password file <none>
I20260811 12:39:21.333961 11167 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:21.334051 11167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:21.334329 11167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:21.337390 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/data/instance:
uuid: "ecc78698274340dca05c58c6ccaaa66c"
format_stamp: "Formatted at 2026-08-11 12:39:21 on dist-test-slave-lgr5"
server_key: "bb9319b478669291dbe6553633bcded5"
server_key_iv: "3a1bee18b3f6b797e05140b661f1856c"
server_key_version: "encryptionkey@0"
I20260811 12:39:21.338181 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/wal/instance:
uuid: "ecc78698274340dca05c58c6ccaaa66c"
format_stamp: "Formatted at 2026-08-11 12:39:21 on dist-test-slave-lgr5"
server_key: "bb9319b478669291dbe6553633bcded5"
server_key_iv: "3a1bee18b3f6b797e05140b661f1856c"
server_key_version: "encryptionkey@0"
I20260811 12:39:21.343374 11167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 12:39:21.346810 11190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:21.348176 11167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:39:21.348331 11167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/wal
uuid: "ecc78698274340dca05c58c6ccaaa66c"
format_stamp: "Formatted at 2026-08-11 12:39:21 on dist-test-slave-lgr5"
server_key: "bb9319b478669291dbe6553633bcded5"
server_key_iv: "3a1bee18b3f6b797e05140b661f1856c"
server_key_version: "encryptionkey@0"
I20260811 12:39:21.348523 11167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:21.375771 11167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:21.380333 11167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:21.380604 11167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:21.398654 11167 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.125:38733
I20260811 12:39:21.398679 11286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.125:38733 every 8 connection(s)
I20260811 12:39:21.400128 11167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
I20260811 12:39:21.404537 11289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:21.409476 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11167
I20260811 12:39:21.409650 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-1/wal/instance
I20260811 12:39:21.410104 3889 external_mini_cluster.cc:1428] Setting key 91b9339e524cb8bbf1cc7f1c1996f4ff
I20260811 12:39:21.412302 11289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:21 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:39:21.434552 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.415421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.3.204.126:40599)
0811 12:39:21.416042 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:21.416048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:21.417071 (+ 1023us) server_negotiation.cc:407] Connection header received
0811 12:39:21.419037 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:21.419043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:21.419126 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:21.419269 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:21.420924 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.421919 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.423588 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.424229 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.424366 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.428485 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:21.428490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:21.428494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:21.428541 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:21.431201 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.432158 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.432162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.432164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.432288 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.432743 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.432746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.432748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.433012 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:21.433131 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:21.434091 (+ 960us) server_negotiation.cc:299] Negotiation successful
0811 12:39:21.434274 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":342,"threads_started":1}
I20260811 12:39:21.436046 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.416049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40599 (local address 127.0.0.1:48950)
0811 12:39:21.416861 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.416945 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:21.418754 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:21.419482 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:21.419490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:21.420043 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:21.420691 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.420712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.422129 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.422134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.423427 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.423441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.424485 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.424491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.424764 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.425475 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:21.425504 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:21.428215 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:21.431362 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:21.431372 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:21.431400 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:21.432023 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:21.432406 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:21.432411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:21.432415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:21.432650 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:21.433132 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:21.433141 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:21.433653 (+ 512us) client_negotiation.cc:769] Sending connection context
0811 12:39:21.433968 (+ 315us) client_negotiation.cc:240] Negotiation successful
0811 12:39:21.434524 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":310,"threads_started":1}
I20260811 12:39:21.437963 11289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.204.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.124@KRBTEST.COM" created.
Aug 11 12:39:21 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:39:21.454078 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.438800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40464)
0811 12:39:21.438990 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.439013 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:21.439120 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:21.440247 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:21.440252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:21.440740 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:21.440942 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.440955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.442364 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.442369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.443694 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.443709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.444590 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.444594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.444736 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.445311 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:21.445331 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:21.447863 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:21.451597 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:21.451601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:21.451604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:21.452016 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:21.452445 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:21.452449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:21.452451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:21.452618 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:21.453342 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:21.453346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:21.453485 (+ 139us) client_negotiation.cc:769] Sending connection context
0811 12:39:21.453596 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 12:39:21.453812 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 12:39:21.454427 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.439075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.3.204.125:38733)
0811 12:39:21.439734 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:21.439742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:21.439775 (+ 33us) server_negotiation.cc:407] Connection header received
0811 12:39:21.439838 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:21.439847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:21.440051 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:21.440214 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:21.441127 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.442208 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.443824 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.444436 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.444554 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.448153 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:21.448172 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:21.448193 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:21.448233 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:21.451446 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.452188 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.452196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.452199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.452335 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.452716 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.452720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.452722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.453193 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:21.453394 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:21.453765 (+ 371us) server_negotiation.cc:299] Negotiation successful
0811 12:39:21.454019 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":367,"threads_started":1}
I20260811 12:39:21.456851 11289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:39:21.458134 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.457598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60250)
0811 12:39:21.457776 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.457936 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 12:39:21.458391 11194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
I20260811 12:39:21.459982 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.459602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60266)
0811 12:39:21.459759 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.459852 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 12:39:21.460219 11289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab.
Entry for principal kudu/127.3.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab.
I20260811 12:39:21.487534 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.124:35029
--webserver_interface=127.3.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab
--principal=kudu/127.3.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:40599,127.3.204.125:38733,127.3.204.124:35029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:39:21.509560 11289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 1
I20260811 12:39:21.510746 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.510247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60274)
0811 12:39:21.510474 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.510585 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 12:39:21.511818 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.511453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60288)
0811 12:39:21.511616 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.511693 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 12:39:21.512048 11289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
I20260811 12:39:21.566828 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } attempt: 5
Aug 11 12:39:21 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:39:21.580281 11289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 2
I20260811 12:39:21.581418 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.567698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.3.204.125:38733)
0811 12:39:21.567901 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:21.567906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:21.567979 (+ 73us) server_negotiation.cc:407] Connection header received
0811 12:39:21.568202 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:21.568207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:21.568290 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:21.568386 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:21.569200 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.569899 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.571089 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.571523 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.571635 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.574828 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:21.574837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:21.574842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:21.574889 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:21.578527 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.579270 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.579277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.579280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.579450 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.579808 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.579814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.579816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.580248 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:21.580408 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:21.580857 (+ 449us) server_negotiation.cc:299] Negotiation successful
0811 12:39:21.581107 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260811 12:39:21.581416 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.567630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40466)
0811 12:39:21.567867 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.567890 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:21.568049 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:21.568443 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:21.568448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:21.568861 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:21.569043 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.569053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.570032 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.570037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.570936 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.570949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.571663 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.571669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.571892 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.572563 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:21.572583 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:21.574592 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:21.578734 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:21.578740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:21.578743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:21.579140 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:21.579564 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:21.579568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:21.579570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:21.579703 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:21.580386 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:21.580391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:21.580579 (+ 188us) client_negotiation.cc:769] Sending connection context
0811 12:39:21.580727 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 12:39:21.581065 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260811 12:39:21.581944 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.580894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60292)
0811 12:39:21.581688 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.581783 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":685}
I20260811 12:39:21.582584 11151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:39:21.582974 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.582584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60306)
0811 12:39:21.582747 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.582841 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 12:39:21.583491 11289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
I20260811 12:39:21.583693 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.583226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60320)
0811 12:39:21.583415 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.583524 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 12:39:21.584679 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.584301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60336)
0811 12:39:21.584449 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.584553 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 12:39:21.584892 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
I20260811 12:39:21.636155 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 1
I20260811 12:39:21.637415 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.636909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60350)
0811 12:39:21.637132 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.637231 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260811 12:39:21.638446 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.638106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60360)
0811 12:39:21.638258 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.638333 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 12:39:21.638674 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
I20260811 12:39:21.674734 11289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 3
I20260811 12:39:21.676290 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.675534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60362)
0811 12:39:21.675878 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.676019 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260811 12:39:21.677518 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.677038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60378)
0811 12:39:21.677280 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.677376 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260811 12:39:21.677743 11289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
I20260811 12:39:21.706882 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 2
I20260811 12:39:21.708047 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.707550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60382)
0811 12:39:21.707767 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.707859 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 12:39:21.709084 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.708735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60384)
0811 12:39:21.708890 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.708966 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 12:39:21.709278 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
I20260811 12:39:21.815505 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 3
I20260811 12:39:21.816638 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.816181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60390)
0811 12:39:21.816376 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.816470 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 12:39:21.817723 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.817281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60398)
0811 12:39:21.817520 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.817615 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260811 12:39:21.817941 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
I20260811 12:39:21.820989 11289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 4
I20260811 12:39:21.822037 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.821567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60400)
0811 12:39:21.821789 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.821876 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260811 12:39:21.823011 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.822667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60412)
0811 12:39:21.822826 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.822902 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 12:39:21.823261 11289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:35029: Network error: Client connection negotiation failed: client connection to 127.3.204.124:35029: connect: Connection refused (error 111)
W20260811 12:39:21.837738 11303 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 12:39:21.838138 11303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:21.838225 11303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:21.838308 11303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:21.845698 11303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:21.845811 11303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:21.845870 11303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:21.845920 11303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:21.845985 11303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:21.856401 11303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:40599,127.3.204.125:38733,127.3.204.124:35029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab
--principal=kudu/127.3.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.124:35029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:21.857779 11303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:21.859233 11303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:21.872022 11320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:21.871989 11318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:21.871941 11316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:21.872702 11303 server_base.cc:1034] running on GCE node
I20260811 12:39:21.873564 11303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:21.875020 11303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:21.876214 11303 hybrid_clock.cc:648] HybridClock initialized: now 1786451961876138 us; error 46 us; skew 500 ppm
I20260811 12:39:21.876567 11303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:21 dist-test-slave-lgr5 krb5kdc[11021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:21.879706 11303 init.cc:377] Logged in from keytab as kudu/127.3.204.124@KRBTEST.COM (short username kudu)
I20260811 12:39:21.881243 11303 webserver.cc:460] Webserver started at http://127.3.204.124:34043/ using document root <none> and password file <none>
I20260811 12:39:21.881945 11303 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:21.882047 11303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:21.882356 11303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:21.885468 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/data/instance:
uuid: "777787a4ccda4433a7c3ba009da28dd2"
format_stamp: "Formatted at 2026-08-11 12:39:21 on dist-test-slave-lgr5"
server_key: "7a56fcbdc57897b2a82856b274d88346"
server_key_iv: "8a283bf3eccde17bbefc3811bf6e24e1"
server_key_version: "encryptionkey@0"
I20260811 12:39:21.886281 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/wal/instance:
uuid: "777787a4ccda4433a7c3ba009da28dd2"
format_stamp: "Formatted at 2026-08-11 12:39:21 on dist-test-slave-lgr5"
server_key: "7a56fcbdc57897b2a82856b274d88346"
server_key_iv: "8a283bf3eccde17bbefc3811bf6e24e1"
server_key_version: "encryptionkey@0"
I20260811 12:39:21.891505 11303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:39:21.894972 11330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:21.896523 11303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 12:39:21.896713 11303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/wal
uuid: "777787a4ccda4433a7c3ba009da28dd2"
format_stamp: "Formatted at 2026-08-11 12:39:21 on dist-test-slave-lgr5"
server_key: "7a56fcbdc57897b2a82856b274d88346"
server_key_iv: "8a283bf3eccde17bbefc3811bf6e24e1"
server_key_version: "encryptionkey@0"
I20260811 12:39:21.896970 11303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:21.923367 11303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:21.929095 11303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:21.929411 11303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:21.956825 11303 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.124:35029
I20260811 12:39:21.956848 11398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.124:35029 every 8 connection(s)
I20260811 12:39:21.958278 11303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
I20260811 12:39:21.962431 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11303
I20260811 12:39:21.962626 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786451585252647-3889-0/minicluster-data/master-2/wal/instance
I20260811 12:39:21.963171 3889 external_mini_cluster.cc:1428] Setting key 507cd697ef52bd9882027c985ef2a96c
I20260811 12:39:21.963439 11400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:21.975092 11400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:21 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
Aug 11 12:39:21 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:39:21.991547 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.965202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.3.204.124:35029)
0811 12:39:21.965989 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:21.966001 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:21.966046 (+ 45us) server_negotiation.cc:407] Connection header received
0811 12:39:21.969063 (+ 3017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:21.969090 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:21.969523 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:21.969897 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:21.971680 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.973714 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.975655 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.976734 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.976911 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.982568 (+ 5657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:21.982579 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:21.982596 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:21.982644 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:21.986647 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.987524 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.987533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.987542 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.987754 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.988129 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.988134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.988137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.988782 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:21.989086 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:21.989597 (+ 511us) server_negotiation.cc:299] Negotiation successful
0811 12:39:21.990044 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
I20260811 12:39:21.995146 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 4
I20260811 12:39:21.998185 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.978963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.3.204.126:40599)
0811 12:39:21.979507 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:21.979513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:21.982877 (+ 3364us) server_negotiation.cc:407] Connection header received
0811 12:39:21.982948 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:21.982954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:21.983036 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:21.983137 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:21.984890 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.985848 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.987493 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.988068 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.988228 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.991690 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:21.991698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:21.991704 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:21.991759 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:21.994828 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.995797 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.995802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.995805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.995964 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:21.996462 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:21.996465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:21.996467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:21.996815 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:21.996973 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:21.997529 (+ 556us) server_negotiation.cc:299] Negotiation successful
0811 12:39:21.997749 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":257,"threads_started":1}
I20260811 12:39:21.998219 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.978520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:40599 (local address 127.0.0.1:48954)
0811 12:39:21.982477 (+ 3957us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.982542 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:21.982738 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:21.983367 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:21.983375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:21.984334 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:21.984701 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.984717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.986016 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.986021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.987320 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.987336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.988265 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.988270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.988414 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:21.988986 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:21.989004 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:21.991471 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:21.995022 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:21.995033 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:21.995039 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:21.995636 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:21.996157 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:21.996161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:21.996164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:21.996335 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:21.996951 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:21.996959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:21.997186 (+ 227us) client_negotiation.cc:769] Sending connection context
0811 12:39:21.997414 (+ 228us) client_negotiation.cc:240] Negotiation successful
0811 12:39:21.997809 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3651,"thread_start_us":3487,"threads_started":1}
I20260811 12:39:21.999835 11400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:21 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
Aug 11 12:39:22 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
Aug 11 12:39:22 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:39:22.011055 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.995939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.3.204.124:35029)
0811 12:39:21.996359 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:21.996364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:21.996381 (+ 17us) server_negotiation.cc:407] Connection header received
0811 12:39:21.996427 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:21.996432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:21.996500 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:21.996594 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:21.997387 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.998454 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.000016 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.000749 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.001418 (+ 669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.004429 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.004434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.004437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.004469 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.007966 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.008858 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.008863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.008867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.009000 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.009498 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.009503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.009507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.009868 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.010016 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.010483 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.010716 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260811 12:39:22.011071 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.995755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60430)
0811 12:39:21.996003 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:21.996022 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:21.996137 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:21.996633 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:21.996637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:21.997104 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:21.997248 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.997256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.998624 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:21.998630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:21.999843 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:21.999858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.000928 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.000934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:22.001118 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.001723 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:22.001741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:22.004229 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:22.008156 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:22.008162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:22.008165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:22.008668 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:22.009127 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:22.009132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:22.009135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:22.009342 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:22.010005 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:22.010011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:22.010192 (+ 181us) client_negotiation.cc:769] Sending connection context
0811 12:39:22.010370 (+ 178us) client_negotiation.cc:240] Negotiation successful
0811 12:39:22.010666 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260811 12:39:22.013515 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:21.995076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.3.204.126:40599)
0811 12:39:21.995813 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:21.995818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:21.995839 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:39:21.995887 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:21.995891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:21.995958 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:21.996163 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:21.997751 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:21.999261 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.001021 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.001684 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.002406 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.006920 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.006929 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.006934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.006977 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.010412 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.011283 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.011288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.011291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.011466 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.011870 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.011875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.011878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.012199 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.012479 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.012912 (+ 433us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.013124 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":383,"threads_started":1}
I20260811 12:39:22.015784 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.000542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:38733 (local address 127.0.0.1:40476)
0811 12:39:22.000807 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:22.000823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:22.000923 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:22.001447 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:22.001452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:22.001945 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:22.002121 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:22.002133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.003526 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.003532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:22.005173 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:22.005187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.005948 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.005954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:22.006170 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.006647 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:22.006668 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:22.008800 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:22.012907 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:22.012912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:22.012915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:22.013425 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:22.013904 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:22.013909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:22.013912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:22.014071 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:22.014822 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:22.014827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:22.014966 (+ 139us) client_negotiation.cc:769] Sending connection context
0811 12:39:22.015121 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 12:39:22.015410 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260811 12:39:22.015880 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.000631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.3.204.125:38733)
0811 12:39:22.000808 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.000814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.000872 (+ 58us) server_negotiation.cc:407] Connection header received
0811 12:39:22.001074 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.001080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.001164 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.001230 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.002301 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.003369 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.005313 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.005788 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.005913 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.009000 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.009005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.009008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.009044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.012738 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.013570 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.013575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.013578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.013745 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.014191 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.014197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.014201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.014698 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.014848 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.015233 (+ 385us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.015453 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 12:39:22.017251 11400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:39:22.017670 11151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa: Bootstrap starting.
I20260811 12:39:22.021571 11151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:22.022878 11151 log.cc:824] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa: Log is configured to *not* fsync() on all Append() calls
Aug 11 12:39:22 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
I20260811 12:39:22.026427 11151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa: No bootstrap required, opened a new log
Aug 11 12:39:22 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:39:22.032295 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.018079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60432)
0811 12:39:22.018261 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:22.018291 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:22.018418 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:22.018910 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:22.018915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:22.019422 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:22.019607 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:22.019618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.021002 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.021007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:22.022240 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:22.022251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.022901 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.022907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:22.023088 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.023649 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:22.023672 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:22.026367 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:22.029727 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:22.029732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:22.029737 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:22.030091 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:22.030450 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:22.030456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:22.030460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:22.030622 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:22.031245 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:22.031251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:22.031401 (+ 150us) client_negotiation.cc:769] Sending connection context
0811 12:39:22.031591 (+ 190us) client_negotiation.cc:240] Negotiation successful
0811 12:39:22.031919 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 12:39:22.032713 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.018482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.3.204.124:35029)
0811 12:39:22.018641 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.018646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.018663 (+ 17us) server_negotiation.cc:407] Connection header received
0811 12:39:22.018707 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.018711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.018777 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.018887 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.019764 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.020834 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.022357 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.022757 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.022861 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.026628 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.026635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.026639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.026677 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.029622 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.030198 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.030203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.030207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.030333 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.030738 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.030742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.030745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.031116 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.031261 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.031724 (+ 463us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.032358 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":382,"server-negotiator.queue_time_us":55}
I20260811 12:39:22.036165 11151 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } }
I20260811 12:39:22.036505 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.015746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.3.204.125:38733)
0811 12:39:22.016194 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.016199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.020322 (+ 4123us) server_negotiation.cc:407] Connection header received
0811 12:39:22.020383 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.020391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.020466 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.020557 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.022102 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.023214 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.024659 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.025194 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.025309 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.030445 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.030452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.030456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.030489 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.033881 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.034538 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.034545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.034549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.034681 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.035061 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.035067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.035070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.035342 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.035583 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.035891 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.036168 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":4829,"threads_started":1}
I20260811 12:39:22.036578 11151 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a57e25614543dba255a41aff47d1fa, State: Initialized, Role: FOLLOWER
I20260811 12:39:22.037108 11151 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } }
I20260811 12:39:22.038259 11414 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.038479 11400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2: Bootstrap starting.
I20260811 12:39:22.038609 11414 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:22.039968 11151 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:22.041218 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:22.042618 11400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:22.044001 11400 log.cc:824] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:22.044613 11107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d6a57e25614543dba255a41aff47d1fa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48956
I20260811 12:39:22.045161 11107 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 12:39:22.045794 11107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:22.047917 11400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2: No bootstrap required, opened a new log
I20260811 12:39:22.048313 11107 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } }
I20260811 12:39:22.049768 11107 leader_election.cc:288] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [CANDIDATE]: Term 1 election: Requested vote from peers ecc78698274340dca05c58c6ccaaa66c (127.3.204.125:38733), 777787a4ccda4433a7c3ba009da28dd2 (127.3.204.124:35029)
I20260811 12:39:22.058236 11400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } }
I20260811 12:39:22.058795 11400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777787a4ccda4433a7c3ba009da28dd2, State: Initialized, Role: FOLLOWER
I20260811 12:39:22.059597 11400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } }
I20260811 12:39:22.054766 11044 master_runner.cc:186] Error getting master registration for 127.3.204.125:38733: OK, instance_id { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" instance_seqno: 1786451961393411 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 12:39:22.077381 11044 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 12:39:22.078457 11400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:22.078591 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6a57e25614543dba255a41aff47d1fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "777787a4ccda4433a7c3ba009da28dd2"
I20260811 12:39:22.079166 11372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [term 0 FOLLOWER]: Advancing to term 1
W20260811 12:39:22.079878 11436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 12:39:22.079996 11436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 12:39:22.082944 11372 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a57e25614543dba255a41aff47d1fa in term 1.
I20260811 12:39:22.083820 11072 leader_election.cc:302] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 777787a4ccda4433a7c3ba009da28dd2, d6a57e25614543dba255a41aff47d1fa; no voters:
I20260811 12:39:22.084367 11414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [term 1 FOLLOWER]: Leader election won for term 1
W20260811 12:39:22.096518 11077 tablet.cc:2369] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:39:22.076920 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.098843 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:22.104769 11414 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [term 1 LEADER]: Becoming Leader. State: Replica: d6a57e25614543dba255a41aff47d1fa, State: Running, Role: LEADER
I20260811 12:39:22.105005 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.055653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.3.204.126:40599)
0811 12:39:22.055880 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.055887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.077752 (+ 21865us) server_negotiation.cc:407] Connection header received
0811 12:39:22.078001 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.078007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.078142 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.078254 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.080530 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.081751 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.086267 (+ 4516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.087052 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.087199 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.092984 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.092993 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.092998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.093052 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.098722 (+ 5670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.100269 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.100277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.100281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.100596 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.101470 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.101475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.101478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.102161 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.102638 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.103892 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.104421 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 12:39:22.105428 11414 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } }
I20260811 12:39:22.114585 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.061433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.3.204.124:35029)
0811 12:39:22.061628 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.061635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.078499 (+ 16864us) server_negotiation.cc:407] Connection header received
0811 12:39:22.083933 (+ 5434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.083941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.084065 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.084166 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.086296 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.087448 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.089511 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.090290 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.090409 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.105413 (+ 15004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.105422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.105428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.105481 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.109806 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.111302 (+ 1496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.111310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.111313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.111483 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.112297 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.112302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.112306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.112687 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.112964 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.113874 (+ 910us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.114180 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 12:39:22.119817 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.056152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.3.204.125:38733)
0811 12:39:22.056328 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.056333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.070614 (+ 14281us) server_negotiation.cc:407] Connection header received
0811 12:39:22.071163 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.071170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.071256 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.071343 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.094194 (+ 22851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.095491 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.099755 (+ 4264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.100543 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.100664 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.111180 (+ 10516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.111189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.111195 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.111249 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.115141 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.116467 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.116473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.116476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.116684 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.117827 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.117833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.117836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.118232 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.118495 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.118932 (+ 437us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.119324 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 12:39:22.122403 11289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } attempt: 5
Aug 11 12:39:22 dist-test-slave-lgr5 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
I20260811 12:39:22.134639 11452 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a57e25614543dba255a41aff47d1fa. Latest consensus state: current_term: 1 leader_uuid: "d6a57e25614543dba255a41aff47d1fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.135604 11452 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:22.148708 11467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:22.149681 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.122979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:35029 (local address 127.0.0.1:60454)
0811 12:39:22.123234 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:22.123253 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:22.123353 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:22.123769 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:22.123774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:22.124338 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:22.124550 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:22.124563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.126069 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.126075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:22.127478 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:22.127495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.128562 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.128569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:22.128984 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.129741 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:22.129768 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:22.133116 (+ 3348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:22.142409 (+ 9293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:22.142417 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:22.142422 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:22.143137 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:22.143651 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:22.143657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:22.143661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:22.147479 (+ 3818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:22.148441 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:22.148448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:22.148642 (+ 194us) client_negotiation.cc:769] Sending connection context
0811 12:39:22.148813 (+ 171us) client_negotiation.cc:240] Negotiation successful
0811 12:39:22.149211 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260811 12:39:22.150902 11467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 12:39:22.151022 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.123204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.3.204.124:35029)
0811 12:39:22.123392 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.123398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.123419 (+ 21us) server_negotiation.cc:407] Connection header received
0811 12:39:22.123535 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.123540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.123629 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.123739 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.124694 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.125884 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.127634 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.128410 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.128540 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.133375 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:22.133382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:22.133388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:22.133436 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:22.142173 (+ 8737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.143311 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.143318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.143323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.143512 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:22.147664 (+ 4152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:22.147670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:22.147675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:22.148161 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:22.148364 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.150212 (+ 1848us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.150547 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 12:39:22.160845 11289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c: Bootstrap starting.
I20260811 12:39:22.162153 11372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [term 1 FOLLOWER]: Refusing update from remote peer d6a57e25614543dba255a41aff47d1fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:39:22.163369 11452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:39:22.169384 11289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:22.171271 11289 log.cc:824] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:22.179674 11289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c: No bootstrap required, opened a new log
I20260811 12:39:22.181010 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a57e25614543dba255a41aff47d1fa. Latest consensus state: current_term: 1 leader_uuid: "d6a57e25614543dba255a41aff47d1fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.181324 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 12:39:22.181942 11339 tablet.cc:2369] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:39:22.186419 11475 mvcc.cc:205] Tried to move back new op lower bound from 7317307237003296768 to 7317307236789854208. Current Snapshot: MvccSnapshot[applied={T|T < 7317307237003296768}]
I20260811 12:39:22.193789 11289 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } }
I20260811 12:39:22.194416 11289 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc78698274340dca05c58c6ccaaa66c, State: Initialized, Role: FOLLOWER
I20260811 12:39:22.195103 11289 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } }
I20260811 12:39:22.196580 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6a57e25614543dba255a41aff47d1fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.196856 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:22.201156 11414 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6a57e25614543dba255a41aff47d1fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.201496 11414 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:22.205919 11289 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:22.205984 11237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6a57e25614543dba255a41aff47d1fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ecc78698274340dca05c58c6ccaaa66c"
I20260811 12:39:22.206501 11237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:22.203810 11487 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.207891 11487 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:22.210333 11237 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a57e25614543dba255a41aff47d1fa in term 1.
I20260811 12:39:22.211400 11233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [term 1 FOLLOWER]: Refusing update from remote peer d6a57e25614543dba255a41aff47d1fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:39:22.212749 11467 catalog_manager.cc:1347] Generated new cluster ID: 27d480ada62c45a3895d3e67e80b0d9d
I20260811 12:39:22.212875 11467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:22.213915 11452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:39:22.229785 11467 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:22.231515 11467 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:22.227469 11303 master_runner.cc:186] Error getting master registration for 127.3.204.125:38733: OK, instance_id { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" instance_seqno: 1786451961393411 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 12:39:22.238339 11303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 12:39:22.247849 11480 catalog_manager.cc:1259] Loaded cluster ID: 27d480ada62c45a3895d3e67e80b0d9d
I20260811 12:39:22.252211 11467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa: Generated new TSK 0
I20260811 12:39:22.253175 11467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:22.258033 11414 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6a57e25614543dba255a41aff47d1fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.258345 11414 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a57e25614543dba255a41aff47d1fa [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:22.265161 11487 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a57e25614543dba255a41aff47d1fa. Latest consensus state: current_term: 1 leader_uuid: "d6a57e25614543dba255a41aff47d1fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.265550 11487 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:22.265933 11500 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6a57e25614543dba255a41aff47d1fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a57e25614543dba255a41aff47d1fa" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 40599 } } peers { permanent_uuid: "ecc78698274340dca05c58c6ccaaa66c" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 38733 } } peers { permanent_uuid: "777787a4ccda4433a7c3ba009da28dd2" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 35029 } } }
I20260811 12:39:22.266934 11500 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:22.269101 11480 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2: loading cluster ID for follower catalog manager: success
I20260811 12:39:22.281528 11480 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2: acquiring CA information for follower catalog manager: success
I20260811 12:39:22.286067 11480 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 777787a4ccda4433a7c3ba009da28dd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 12:39:22.303961 11519 catalog_manager.cc:1259] Loaded cluster ID: 27d480ada62c45a3895d3e67e80b0d9d
I20260811 12:39:22.304303 11519 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c: loading cluster ID for follower catalog manager: success
I20260811 12:39:22.308549 11519 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c: acquiring CA information for follower catalog manager: success
I20260811 12:39:22.311308 11519 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecc78698274340dca05c58c6ccaaa66c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 12:39:22.369402 3889 mini_kdc.cc:348] Time spent kdestroy: real 0.101s user 0.005s sys 0.027s
W20260811 12:39:22.430053 11299 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:22.403676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.3.204.125:38733)
0811 12:39:22.403886 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.403893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.426144 (+ 22251us) server_negotiation.cc:407] Connection header received
0811 12:39:22.428190 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.428197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.428297 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.428374 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.429782 (+ 1408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40490: BlockingRecv error: recv got EOF from 127.0.0.1:40490 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260811 12:39:22.430490 11406 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:22.409987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.3.204.124:35029)
0811 12:39:22.410218 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.410229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.428663 (+ 18434us) server_negotiation.cc:407] Connection header received
0811 12:39:22.428928 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.428934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.429021 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.429093 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.430236 (+ 1143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60462: BlockingRecv error: recv got EOF from 127.0.0.1:60462 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 12:39:22.432796 11410 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:22.422253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.3.204.126:40599)
0811 12:39:22.422449 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.422456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.430883 (+ 8427us) server_negotiation.cc:407] Connection header received
0811 12:39:22.431394 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.431400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.431483 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.431557 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:22.432522 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48974: BlockingRecv error: recv got EOF from 127.0.0.1:48974 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 12:39:22.487946 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.468787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.3.204.125:38733)
0811 12:39:22.468972 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.468978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.479864 (+ 10886us) server_negotiation.cc:407] Connection header received
0811 12:39:22.480481 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.480488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.480706 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.480839 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:22.481582 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.482706 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.484720 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.485437 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.485550 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.486344 (+ 794us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:22.487089 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:22.487214 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.487476 (+ 262us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.487587 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 12:39:22.541298 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.525013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.3.204.124:35029)
0811 12:39:22.525220 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.525227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.532791 (+ 7564us) server_negotiation.cc:407] Connection header received
0811 12:39:22.533402 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.533409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.533665 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.533830 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:22.534613 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.535871 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.538014 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.538705 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.538817 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.539636 (+ 819us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:22.540430 (+ 794us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:22.540557 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.540800 (+ 243us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.540904 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 12:39:22.621191 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.576790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.3.204.125:38733)
0811 12:39:22.577046 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.577054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.599685 (+ 22631us) server_negotiation.cc:407] Connection header received
0811 12:39:22.602320 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.602327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.602466 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.602545 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:22.605287 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.606461 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.616393 (+ 9932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.617154 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.617278 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.620221 (+ 2943us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:22.620388 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:22.620471 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.620718 (+ 247us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.620822 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 12:39:22.626168 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.583794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.3.204.124:35029)
0811 12:39:22.583997 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.584003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.601941 (+ 17938us) server_negotiation.cc:407] Connection header received
0811 12:39:22.604180 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.604186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.604283 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.604356 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:22.610038 (+ 5682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.611142 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.618177 (+ 7035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.618974 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.619090 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.625106 (+ 6016us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:22.625292 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:22.625378 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.625726 (+ 348us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.625825 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 12:39:22.629918 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:22.593968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.3.204.126:40599)
0811 12:39:22.594196 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:22.594204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:22.603838 (+ 9634us) server_negotiation.cc:407] Connection header received
0811 12:39:22.608654 (+ 4816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:22.608661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:22.608934 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:22.609126 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:22.613187 (+ 4061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.614387 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.623600 (+ 9213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:22.624277 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:22.624392 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:22.628537 (+ 4145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:22.629036 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:22.629148 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:22.629480 (+ 332us) server_negotiation.cc:299] Negotiation successful
0811 12:39:22.629578 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 12:39:22.636773 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11044
I20260811 12:39:22.638051 11145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:22.766878 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11044
I20260811 12:39:22.801731 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11167
I20260811 12:39:22.802948 11276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:22.926959 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11167
I20260811 12:39:22.955271 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11303
I20260811 12:39:22.956410 11393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:23.064308 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11303
2026-08-11T12:39:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4287 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 12:39:23 dist-test-slave-lgr5 krb5kdc[11579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 12:39:23 dist-test-slave-lgr5 krb5kdc[11579](info): set up 2 sockets
Aug 11 12:39:23 dist-test-slave-lgr5 krb5kdc[11579](info): commencing operation
krb5kdc: starting...
W20260811 12:39:24.357705 3889 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.005s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/test-user.keytab.
Aug 11 12:39:24 dist-test-slave-lgr5 krb5kdc[11579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T12:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T12:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
Entry for principal kudu/127.3.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab.
I20260811 12:39:24.683599 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46287
--webserver_interface=127.3.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:46287,127.3.204.125:37111,127.3.204.124:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 12:39:25.035447 11603 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 12:39:25.035849 11603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:25.035953 11603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:25.036104 11603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:25.043638 11603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:25.043767 11603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:25.043852 11603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:25.043920 11603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:25.044013 11603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:25.054513 11603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:46287,127.3.204.125:37111,127.3.204.124:41839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.126.keytab
--principal=kudu/127.3.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.126:46287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:25.055914 11603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:25.057304 11603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:25.069492 11609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:25.069468 11614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:25.070377 11603 server_base.cc:1034] running on GCE node
W20260811 12:39:25.069537 11610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:25.071519 11603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:25.073056 11603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:25.074280 11603 hybrid_clock.cc:648] HybridClock initialized: now 1786451965074212 us; error 35 us; skew 500 ppm
I20260811 12:39:25.074609 11603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:25 dist-test-slave-lgr5 krb5kdc[11579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:25.077976 11603 init.cc:377] Logged in from keytab as kudu/127.3.204.126@KRBTEST.COM (short username kudu)
I20260811 12:39:25.079614 11603 webserver.cc:460] Webserver started at http://127.3.204.126:33889/ using document root <none> and password file <none>
I20260811 12:39:25.080453 11603 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:25.080564 11603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:25.080929 11603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:25.084270 11603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/data/instance:
uuid: "2612191edaaf4ba1aa9d3010af7e57f7"
format_stamp: "Formatted at 2026-08-11 12:39:25 on dist-test-slave-lgr5"
server_key: "40b0a5ce2e5be234b86cb4330c039d1d"
server_key_iv: "c075a3b0774a944a39153c0fd3d64958"
server_key_version: "encryptionkey@0"
I20260811 12:39:25.085146 11603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/wal/instance:
uuid: "2612191edaaf4ba1aa9d3010af7e57f7"
format_stamp: "Formatted at 2026-08-11 12:39:25 on dist-test-slave-lgr5"
server_key: "40b0a5ce2e5be234b86cb4330c039d1d"
server_key_iv: "c075a3b0774a944a39153c0fd3d64958"
server_key_version: "encryptionkey@0"
I20260811 12:39:25.091132 11603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 12:39:25.095178 11622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:25.096704 11603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:39:25.096900 11603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/wal
uuid: "2612191edaaf4ba1aa9d3010af7e57f7"
format_stamp: "Formatted at 2026-08-11 12:39:25 on dist-test-slave-lgr5"
server_key: "40b0a5ce2e5be234b86cb4330c039d1d"
server_key_iv: "c075a3b0774a944a39153c0fd3d64958"
server_key_version: "encryptionkey@0"
I20260811 12:39:25.097132 11603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:25.123971 11603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:25.128123 11603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:25.128389 11603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:25.147631 11603 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.126:46287
I20260811 12:39:25.147680 11718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.126:46287 every 8 connection(s)
I20260811 12:39:25.149183 11603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/data/info.pb
I20260811 12:39:25.153302 11720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:25.156821 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11603
I20260811 12:39:25.156991 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-0/wal/instance
I20260811 12:39:25.157404 3889 external_mini_cluster.cc:1428] Setting key 6a9a8fe40471c81e92469e192629b737
I20260811 12:39:25.160390 11720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:25 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:39:25.183387 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.163839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.3.204.126:46287)
0811 12:39:25.164713 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:25.164729 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:25.165197 (+ 468us) server_negotiation.cc:407] Connection header received
0811 12:39:25.167249 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:25.167255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:25.167505 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:25.167698 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:25.169119 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.170434 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.171868 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.172514 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.172705 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:25.176214 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:25.176223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:25.176237 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:25.176272 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:25.178920 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:25.179629 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:25.179637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:25.179641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:25.179821 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:25.180227 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:25.180233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:25.180236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:25.180766 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:25.180983 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:25.181534 (+ 551us) server_negotiation.cc:299] Negotiation successful
0811 12:39:25.181833 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":403,"threads_started":1}
I20260811 12:39:25.183405 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.163843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46287 (local address 127.0.0.1:44296)
0811 12:39:25.165025 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.165068 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:25.166855 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:25.167731 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:25.167737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:25.168538 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:25.168902 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:25.168921 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.170584 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.170588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:25.171721 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:25.171734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.172749 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.172755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:25.172929 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:25.173573 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:25.173598 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:25.175978 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:25.179069 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:25.179081 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:25.179096 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:25.179505 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:25.179924 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:25.179929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:25.179933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:25.180123 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:25.180912 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:25.180919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:25.181140 (+ 221us) client_negotiation.cc:769] Sending connection context
0811 12:39:25.181369 (+ 229us) client_negotiation.cc:240] Negotiation successful
0811 12:39:25.181997 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":126,"spinlock_wait_cycles":18688,"thread_start_us":743,"threads_started":1}
I20260811 12:39:25.186686 11720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.204.125@KRBTEST.COM; defaulting to no policy
I20260811 12:39:25.187849 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.187419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46372)
0811 12:39:25.187603 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.187718 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260811 12:39:25.188170 11628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
I20260811 12:39:25.189707 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.189329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46374)
0811 12:39:25.189501 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.189572 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 12:39:25.189898 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:37111: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab.
Entry for principal kudu/127.3.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab.
I20260811 12:39:25.227180 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.125:37111
--webserver_interface=127.3.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab
--principal=kudu/127.3.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:46287,127.3.204.125:37111,127.3.204.124:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:39:25.239333 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } attempt: 1
I20260811 12:39:25.240782 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.240312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46376)
0811 12:39:25.240514 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.240644 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 12:39:25.241791 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.241418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46392)
0811 12:39:25.241573 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.241649 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 12:39:25.241995 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:37111: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
I20260811 12:39:25.310278 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } attempt: 2
I20260811 12:39:25.311578 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.311077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46404)
0811 12:39:25.311331 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.311436 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260811 12:39:25.312662 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.312271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46406)
0811 12:39:25.312440 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.312517 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 12:39:25.312850 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:37111: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
I20260811 12:39:25.404158 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } attempt: 3
I20260811 12:39:25.405339 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.404873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46412)
0811 12:39:25.405088 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.405188 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 12:39:25.406435 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.406041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46420)
0811 12:39:25.406198 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.406279 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 12:39:25.406631 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:37111: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
I20260811 12:39:25.549916 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } attempt: 4
I20260811 12:39:25.551297 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.550654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46426)
0811 12:39:25.551016 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.551117 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260811 12:39:25.552359 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.551950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46428)
0811 12:39:25.552126 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.552215 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 12:39:25.552567 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.125:37111: Network error: Client connection negotiation failed: client connection to 127.3.204.125:37111: connect: Connection refused (error 111)
W20260811 12:39:25.573179 11730 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 12:39:25.573559 11730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:25.573661 11730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:25.573782 11730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 12:39:25.581282 11730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:25.581399 11730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:25.581485 11730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:25.581548 11730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:25.581640 11730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:25.591710 11730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:46287,127.3.204.125:37111,127.3.204.124:41839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.125.keytab
--principal=kudu/127.3.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.125:37111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:25.593075 11730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:25.594523 11730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:25.606688 11739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:25.606527 11740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:25.606663 11747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:25.607852 11730 server_base.cc:1034] running on GCE node
I20260811 12:39:25.608774 11730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:25.610273 11730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:25.611529 11730 hybrid_clock.cc:648] HybridClock initialized: now 1786451965611455 us; error 49 us; skew 500 ppm
I20260811 12:39:25.611876 11730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:25 dist-test-slave-lgr5 krb5kdc[11579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:25.615195 11730 init.cc:377] Logged in from keytab as kudu/127.3.204.125@KRBTEST.COM (short username kudu)
I20260811 12:39:25.616763 11730 webserver.cc:460] Webserver started at http://127.3.204.125:39423/ using document root <none> and password file <none>
I20260811 12:39:25.617494 11730 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:25.617589 11730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:25.617868 11730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:25.621039 11730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/data/instance:
uuid: "252ea753a6324e0d888cd8b3208eede9"
format_stamp: "Formatted at 2026-08-11 12:39:25 on dist-test-slave-lgr5"
server_key: "16d42b8e4ce4ddff0767e3037c4da867"
server_key_iv: "ebd269d86b33eb2681db60ce6518074d"
server_key_version: "encryptionkey@0"
I20260811 12:39:25.621801 11730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/wal/instance:
uuid: "252ea753a6324e0d888cd8b3208eede9"
format_stamp: "Formatted at 2026-08-11 12:39:25 on dist-test-slave-lgr5"
server_key: "16d42b8e4ce4ddff0767e3037c4da867"
server_key_iv: "ebd269d86b33eb2681db60ce6518074d"
server_key_version: "encryptionkey@0"
I20260811 12:39:25.627130 11730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:39:25.630781 11764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:25.632272 11730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 12:39:25.632431 11730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/wal
uuid: "252ea753a6324e0d888cd8b3208eede9"
format_stamp: "Formatted at 2026-08-11 12:39:25 on dist-test-slave-lgr5"
server_key: "16d42b8e4ce4ddff0767e3037c4da867"
server_key_iv: "ebd269d86b33eb2681db60ce6518074d"
server_key_version: "encryptionkey@0"
I20260811 12:39:25.632628 11730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:25.644569 11730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:25.648973 11730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:25.649288 11730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:25.667248 11730 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.125:37111
I20260811 12:39:25.667268 11826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.125:37111 every 8 connection(s)
I20260811 12:39:25.668673 11730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/data/info.pb
I20260811 12:39:25.670035 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11730
I20260811 12:39:25.670233 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-1/wal/instance
I20260811 12:39:25.670814 3889 external_mini_cluster.cc:1428] Setting key 3cfe01a466cef7d52d4dc9295667824d
I20260811 12:39:25.673446 11827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:25.682301 11827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:25 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.204.124@KRBTEST.COM; defaulting to no policy
I20260811 12:39:25.705533 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.685650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.3.204.126:46287)
0811 12:39:25.686200 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:25.686213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:25.687288 (+ 1075us) server_negotiation.cc:407] Connection header received
0811 12:39:25.689325 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:25.689331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:25.689429 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:25.689551 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:25.691160 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.692290 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.694098 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.694717 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.694847 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:25.698718 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:25.698725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:25.698729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:25.698770 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:25.702253 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:25.703036 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:25.703042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:25.703046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:25.703218 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:25.703587 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:25.703592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:25.703596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:25.703977 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:25.704186 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:25.704908 (+ 722us) server_negotiation.cc:299] Negotiation successful
0811 12:39:25.705158 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":266,"threads_started":1}
I20260811 12:39:25.706413 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.686277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46287 (local address 127.0.0.1:44308)
0811 12:39:25.687088 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.687147 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:25.689027 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:25.689764 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:25.689773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:25.690354 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:25.690930 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:25.690949 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.692505 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.692510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:25.693828 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:25.693910 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.695078 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.695084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:25.695322 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:25.696037 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:25.696063 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:25.698459 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:25.702400 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:25.702407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:25.702422 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:25.702888 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:25.703314 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:25.703317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:25.703319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:25.703484 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:25.704159 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:25.704166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:25.704537 (+ 371us) client_negotiation.cc:769] Sending connection context
0811 12:39:25.704801 (+ 264us) client_negotiation.cc:240] Negotiation successful
0811 12:39:25.705239 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":308,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.204.124@KRBTEST.COM" created.
I20260811 12:39:25.708240 11827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:25 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:39:25.722684 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.709035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46440)
0811 12:39:25.709221 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.709240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:25.709346 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:25.710547 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:25.710550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:25.710889 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:25.711032 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:25.711040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.712823 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.712827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:25.713679 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:25.713690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.714700 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.714704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:25.714822 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:25.715391 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:25.715407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:25.717381 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:25.720421 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:25.720426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:25.720431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:25.720788 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:25.721143 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:25.721148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:25.721152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:25.721288 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:25.721962 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:25.721966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:25.722084 (+ 118us) client_negotiation.cc:769] Sending connection context
0811 12:39:25.722200 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 12:39:25.722407 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 12:39:25.723067 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.709306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.3.204.125:37111)
0811 12:39:25.710034 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:25.710041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:25.710074 (+ 33us) server_negotiation.cc:407] Connection header received
0811 12:39:25.710127 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:25.710133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:25.710374 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:25.710533 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:25.711205 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.712695 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.713814 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.714553 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.714677 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:25.717576 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:25.717587 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:25.717600 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:25.717632 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:25.720301 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:25.720910 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:25.720917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:25.720919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:25.721042 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:25.721393 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:25.721397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:25.721399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:25.721746 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:25.721944 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:25.722380 (+ 436us) server_negotiation.cc:299] Negotiation successful
0811 12:39:25.722638 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":378,"threads_started":1}
I20260811 12:39:25.725666 11827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:39:25.726887 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.726456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35930)
0811 12:39:25.726626 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.726738 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 12:39:25.727198 11766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
I20260811 12:39:25.728845 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.728527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35946)
0811 12:39:25.728658 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.728739 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 12:39:25.729063 11827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab.
Entry for principal kudu/127.3.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab.
I20260811 12:39:25.748062 3889 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
/tmp/dist-test-task1nKHSZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.124:41839
--webserver_interface=127.3.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab
--principal=kudu/127.3.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.204.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.204.126:46287,127.3.204.125:37111,127.3.204.124:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 12:39:25.778420 11827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 1
I20260811 12:39:25.779778 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.779250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35954)
0811 12:39:25.779462 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.779590 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260811 12:39:25.780887 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.780496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35960)
0811 12:39:25.780669 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.780767 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 12:39:25.781122 11827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
I20260811 12:39:25.849375 11827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 2
I20260811 12:39:25.850589 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.850115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35976)
0811 12:39:25.850325 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.850420 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260811 12:39:25.851581 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.851238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35982)
0811 12:39:25.851388 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.851456 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 12:39:25.851792 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } attempt: 5
W20260811 12:39:25.851835 11827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Aug 11 12:39:25 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:39:25.870251 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.852419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.3.204.125:37111)
0811 12:39:25.852570 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:25.852575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:25.852666 (+ 91us) server_negotiation.cc:407] Connection header received
0811 12:39:25.852866 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:25.852869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:25.852950 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:25.853044 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:25.854054 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.854743 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.856456 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.856899 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.857010 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:25.864687 (+ 7677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:25.864693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:25.864697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:25.864737 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:25.867337 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:25.868188 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:25.868192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:25.868194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:25.868332 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:25.868801 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:25.868807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:25.868809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:25.869164 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:25.869307 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:25.869787 (+ 480us) server_negotiation.cc:299] Negotiation successful
0811 12:39:25.869986 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 12:39:25.870373 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.852353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46442)
0811 12:39:25.852570 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.852592 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:25.852748 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:25.853106 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:25.853112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:25.853636 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:25.853868 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:25.853880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.854877 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.854883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:25.856282 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:25.856299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:25.857051 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:25.857058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:25.857292 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:25.861750 (+ 4458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:25.861775 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:25.864409 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:25.867511 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:25.867518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:25.867523 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:25.868047 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:25.868448 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:25.868453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:25.868457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:25.868627 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:25.869297 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:25.869302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:25.869496 (+ 194us) client_negotiation.cc:769] Sending connection context
0811 12:39:25.869645 (+ 149us) client_negotiation.cc:240] Negotiation successful
0811 12:39:25.869989 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260811 12:39:25.871590 11720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:39:25.872941 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.872482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35986)
0811 12:39:25.872693 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.872781 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260811 12:39:25.874068 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.873667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35990)
0811 12:39:25.873848 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.873959 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260811 12:39:25.874277 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
I20260811 12:39:25.925582 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 1
I20260811 12:39:25.926983 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.926458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:35996)
0811 12:39:25.926689 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.926790 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260811 12:39:25.927997 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.927641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36012)
0811 12:39:25.927791 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.927867 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 12:39:25.928231 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
I20260811 12:39:25.943104 11827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 3
I20260811 12:39:25.944391 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.943843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36028)
0811 12:39:25.944107 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.944206 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 12:39:25.945600 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.945123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36040)
0811 12:39:25.945389 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.945474 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260811 12:39:25.945815 11827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
I20260811 12:39:25.996438 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 2
I20260811 12:39:25.997602 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.997108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36046)
0811 12:39:25.997325 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.997421 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 12:39:25.998569 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:25.998247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36062)
0811 12:39:25.998392 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:25.998462 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 12:39:25.998797 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
I20260811 12:39:26.089076 11827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 4
I20260811 12:39:26.090355 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.089768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36066)
0811 12:39:26.090039 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.090154 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260811 12:39:26.091449 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.091055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36074)
0811 12:39:26.091222 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.091313 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 12:39:26.091674 11827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
W20260811 12:39:26.104514 11840 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 12:39:26.104851 11840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 12:39:26.104921 11840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 12:39:26.105013 11840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 12:39:26.104990 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 3
I20260811 12:39:26.106132 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.105658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36088)
0811 12:39:26.105849 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.105944 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 12:39:26.107113 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.106754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36094)
0811 12:39:26.106937 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.107007 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260811 12:39:26.107311 11720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.204.124:41839: Network error: Client connection negotiation failed: client connection to 127.3.204.124:41839: connect: Connection refused (error 111)
W20260811 12:39:26.112998 11840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 12:39:26.113090 11840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 12:39:26.113181 11840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 12:39:26.113245 11840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 12:39:26.113329 11840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 12:39:26.123553 11840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.204.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.204.126:46287,127.3.204.125:37111,127.3.204.124:41839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/krb5kdc/kudu_127.3.204.124.keytab
--principal=kudu/127.3.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.204.124:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 11:11:46 UTC on ccycloud.quasar-zjjjhb.root.comops.site
build id 4580142
ASAN enabled
I20260811 12:39:26.125043 11840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 12:39:26.126364 11840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 12:39:26.138212 11851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:26.138296 11853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 12:39:26.138674 11850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 12:39:26.139137 11840 server_base.cc:1034] running on GCE node
I20260811 12:39:26.140031 11840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 12:39:26.141620 11840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 12:39:26.142863 11840 hybrid_clock.cc:648] HybridClock initialized: now 1786451966142785 us; error 50 us; skew 500 ppm
I20260811 12:39:26.143208 11840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 12:39:26.146464 11840 init.cc:377] Logged in from keytab as kudu/127.3.204.124@KRBTEST.COM (short username kudu)
I20260811 12:39:26.148180 11840 webserver.cc:460] Webserver started at http://127.3.204.124:44971/ using document root <none> and password file <none>
I20260811 12:39:26.148892 11840 fs_manager.cc:359] Metadata directory not provided
I20260811 12:39:26.148998 11840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 12:39:26.149318 11840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 12:39:26.152444 11840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/data/instance:
uuid: "b8322cc2e767463aa323231637e2ed89"
format_stamp: "Formatted at 2026-08-11 12:39:26 on dist-test-slave-lgr5"
server_key: "2388adb04317729383bc073fb203db2e"
server_key_iv: "7bce67e42eb57ff240babc65d06bd8b1"
server_key_version: "encryptionkey@0"
I20260811 12:39:26.153281 11840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/wal/instance:
uuid: "b8322cc2e767463aa323231637e2ed89"
format_stamp: "Formatted at 2026-08-11 12:39:26 on dist-test-slave-lgr5"
server_key: "2388adb04317729383bc073fb203db2e"
server_key_iv: "7bce67e42eb57ff240babc65d06bd8b1"
server_key_version: "encryptionkey@0"
I20260811 12:39:26.158954 11840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 12:39:26.162949 11873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 12:39:26.164450 11840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260811 12:39:26.164646 11840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/data,/tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/wal
uuid: "b8322cc2e767463aa323231637e2ed89"
format_stamp: "Formatted at 2026-08-11 12:39:26 on dist-test-slave-lgr5"
server_key: "2388adb04317729383bc073fb203db2e"
server_key_iv: "7bce67e42eb57ff240babc65d06bd8b1"
server_key_version: "encryptionkey@0"
I20260811 12:39:26.164875 11840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 12:39:26.183064 11840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 12:39:26.187506 11840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 12:39:26.187779 11840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 12:39:26.205189 11840 rpc_server.cc:307] RPC server started. Bound to: 127.3.204.124:41839
I20260811 12:39:26.205210 11954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.204.124:41839 every 8 connection(s)
I20260811 12:39:26.206737 11840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/data/info.pb
I20260811 12:39:26.210666 11956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 12:39:26.211087 3889 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu as pid 11840
I20260811 12:39:26.211270 3889 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1nKHSZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786451585252647-3889-0/minicluster-data/master-2/wal/instance
I20260811 12:39:26.211735 3889 external_mini_cluster.cc:1428] Setting key 09a2879a693d58b9a9962d159829f104
I20260811 12:39:26.219236 11956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:39:26.234866 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.213543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.3.204.124:41839)
0811 12:39:26.214300 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.214312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.214351 (+ 39us) server_negotiation.cc:407] Connection header received
0811 12:39:26.216742 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.216768 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.217220 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.217650 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.219145 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.220520 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.222113 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.223087 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.223267 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.227378 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.227388 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.227400 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.227437 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.230628 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.231458 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.231465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.231472 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.231619 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.232033 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.232037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.232039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.232615 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.232906 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.233365 (+ 459us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.233800 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":319,"threads_started":1}
I20260811 12:39:26.236513 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.221738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.3.204.126:46287)
0811 12:39:26.222258 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.222265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.222700 (+ 435us) server_negotiation.cc:407] Connection header received
0811 12:39:26.222887 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.222890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.222979 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.223074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.224349 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.225017 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.226601 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.227014 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.227124 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.230839 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.230847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.230851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.230893 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.233612 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.234361 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.234366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.234370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.234521 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.234865 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.234870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.234874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.235207 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.235357 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.235820 (+ 463us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.236063 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":232,"threads_started":1}
I20260811 12:39:26.236534 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.221799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.126:46287 (local address 127.0.0.1:44320)
0811 12:39:26.222563 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.222598 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:26.222761 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:26.223166 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:26.223173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:26.223847 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:26.224198 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.224210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.225169 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.225173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.226440 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.226457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.227232 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.227240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.227405 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.228018 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:26.228042 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:26.230606 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:26.233784 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.233797 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.233804 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.234241 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.234629 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.234633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.234635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.234759 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.235336 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:26.235343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:26.235526 (+ 183us) client_negotiation.cc:769] Sending connection context
0811 12:39:26.235683 (+ 157us) client_negotiation.cc:240] Negotiation successful
0811 12:39:26.236022 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":404,"threads_started":1}
I20260811 12:39:26.237996 11956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.126@KRBTEST.COM
I20260811 12:39:26.251020 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.238800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.125:37111 (local address 127.0.0.1:46454)
0811 12:39:26.239028 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.239042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:26.239125 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:26.239394 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:26.239397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:26.239726 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:26.239881 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.239889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.241015 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.241020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.241951 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.241966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.242731 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.242737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.242942 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.243500 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:26.243522 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:26.245535 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:26.248768 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.248774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.248778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.249253 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.249574 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.249579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.249583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.249749 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.250217 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:26.250221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:26.250361 (+ 140us) client_negotiation.cc:769] Sending connection context
0811 12:39:26.250481 (+ 120us) client_negotiation.cc:240] Negotiation successful
0811 12:39:26.250678 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260811 12:39:26.251119 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.238764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.3.204.125:37111)
0811 12:39:26.238972 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.238977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.239101 (+ 124us) server_negotiation.cc:407] Connection header received
0811 12:39:26.239207 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.239212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.239287 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.239392 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.240022 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.240865 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.242098 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.242592 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.242687 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.245753 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.245758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.245761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.245798 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.248640 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.249354 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.249358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.249360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.249473 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.249844 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.249847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.249849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.250092 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.250248 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.250608 (+ 360us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.250808 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 12:39:26.252276 11956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } has no permanent_uuid. Determining permanent_uuid...
I20260811 12:39:26.252744 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.238169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.3.204.126:46287)
0811 12:39:26.238318 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.238321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.238517 (+ 196us) server_negotiation.cc:407] Connection header received
0811 12:39:26.238686 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.238692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.238762 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.238839 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.240356 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.241064 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.242277 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.242674 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.242794 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.247092 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.247097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.247100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.247133 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.250031 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.250758 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.250763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.250766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.250910 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.251338 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.251343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.251346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.251680 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.251969 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.252278 (+ 309us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.252474 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.124@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.204.125@KRBTEST.COM
I20260811 12:39:26.266197 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.253385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.3.204.124:41839)
0811 12:39:26.253525 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.253528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.253541 (+ 13us) server_negotiation.cc:407] Connection header received
0811 12:39:26.253574 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.253577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.253629 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.253726 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.254458 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.255315 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.256906 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.257550 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.257670 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.261296 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.261301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.261305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.261336 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.263773 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.264421 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.264435 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.264439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.264556 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.264885 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.264890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.264894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.265177 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.265318 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.265657 (+ 339us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.265847 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 12:39:26.266376 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.253082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36114)
0811 12:39:26.253254 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.253271 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:26.253363 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:26.253779 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:26.253783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:26.254146 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:26.254322 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.254336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.255446 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.255453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.256779 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.256793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.257695 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.257699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.257941 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.258520 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:26.258543 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:26.261095 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:26.263903 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.263907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.263910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.264316 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.264654 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.264658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.264660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.264778 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.265287 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:26.265291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:26.265423 (+ 132us) client_negotiation.cc:769] Sending connection context
0811 12:39:26.265542 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 12:39:26.266014 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":129}
I20260811 12:39:26.270726 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.254470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.3.204.125:37111)
0811 12:39:26.254642 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.254648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.254668 (+ 20us) server_negotiation.cc:407] Connection header received
0811 12:39:26.254824 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.254900 (+ 76us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.254984 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.255091 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.256688 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.257697 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.259378 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.259826 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.259956 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.264169 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.264174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.264177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.264212 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.266993 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.267732 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.267736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.267739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.267874 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.268365 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.268371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.268374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.268749 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.269027 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.270129 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.270374 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 12:39:26.272686 11956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89: Bootstrap starting.
I20260811 12:39:26.276427 3889 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 12:39:26.276759 11956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:26.277890 11956 log.cc:824] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:26.281312 11956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89: No bootstrap required, opened a new log
I20260811 12:39:26.278851 11668 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2612191edaaf4ba1aa9d3010af7e57f7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44334
I20260811 12:39:26.284504 11720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 4
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.126@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
I20260811 12:39:26.291834 11956 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } }
I20260811 12:39:26.292315 11956 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8322cc2e767463aa323231637e2ed89, State: Initialized, Role: FOLLOWER
I20260811 12:39:26.292883 11956 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } }
I20260811 12:39:26.294232 11968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.294677 11968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:26.295496 11956 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:26.299645 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.285190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36118)
0811 12:39:26.285401 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.285422 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:26.285571 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:26.285916 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:26.285921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:26.286315 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:26.286488 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.286498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.287701 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.287706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.288695 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.288709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.289480 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.289486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.289682 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.290258 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:26.290276 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:26.292472 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:26.296722 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.296728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.296732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.297291 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.297820 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.297825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.297829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.298009 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.298711 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:26.298715 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:26.298879 (+ 164us) client_negotiation.cc:769] Sending connection context
0811 12:39:26.299028 (+ 149us) client_negotiation.cc:240] Negotiation successful
0811 12:39:26.299309 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 12:39:26.300666 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.285259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.3.204.124:41839)
0811 12:39:26.285429 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.285434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.285506 (+ 72us) server_negotiation.cc:407] Connection header received
0811 12:39:26.285688 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.285694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.285787 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.285873 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.286633 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.287545 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.288840 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.289324 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.289438 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.292669 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.292676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.292682 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.292721 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.296550 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.297467 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.297473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.297476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.297666 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.298156 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.298161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.298165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.298588 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.298750 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.300039 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.300305 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:39:26.307004 11720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7: Bootstrap starting.
I20260811 12:39:26.311475 11720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:26.313047 11720 log.cc:824] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7: Log is configured to *not* fsync() on all Append() calls
W20260811 12:39:26.316716 11985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 12:39:26.316859 11985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 12:39:26.317227 11720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7: No bootstrap required, opened a new log
I20260811 12:39:26.307411 11840 master_runner.cc:186] Error getting master registration for 127.3.204.126:46287: OK, instance_id { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" instance_seqno: 1786451965141774 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 12:39:26.318671 11840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 12:39:26.325531 11720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } }
I20260811 12:39:26.325974 11720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2612191edaaf4ba1aa9d3010af7e57f7, State: Initialized, Role: FOLLOWER
I20260811 12:39:26.326503 11720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } }
I20260811 12:39:26.327533 11991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.327934 11991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:26.328819 11720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:26.328902 11668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 12:39:26.329090 11668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:26.332866 11668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } }
I20260811 12:39:26.335753 11912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b8322cc2e767463aa323231637e2ed89"
I20260811 12:39:26.336356 11912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:26.340319 11912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2612191edaaf4ba1aa9d3010af7e57f7 in term 1.
I20260811 12:39:26.340844 11668 leader_election.cc:288] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [CANDIDATE]: Term 1 election: Requested vote from peers 252ea753a6324e0d888cd8b3208eede9 (127.3.204.125:37111), b8322cc2e767463aa323231637e2ed89 (127.3.204.124:41839)
I20260811 12:39:26.341214 11625 leader_election.cc:302] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2612191edaaf4ba1aa9d3010af7e57f7, b8322cc2e767463aa323231637e2ed89; no voters:
I20260811 12:39:26.341966 11991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 12:39:26.344719 11991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [term 1 LEADER]: Becoming Leader. State: Replica: 2612191edaaf4ba1aa9d3010af7e57f7, State: Running, Role: LEADER
I20260811 12:39:26.345325 11991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } }
W20260811 12:39:26.350158 12003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 12:39:26.350296 12003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 12:39:26.359807 12004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2612191edaaf4ba1aa9d3010af7e57f7. Latest consensus state: current_term: 1 leader_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.360199 12004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:26.363867 11603 master_runner.cc:186] Error getting master registration for 127.3.204.125:37111: OK, instance_id { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" instance_seqno: 1786451965662162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 12:39:26.374936 11603 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 12:39:26.375120 12015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 12:39:26.377578 12015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260811 12:39:26.381402 11632 tablet.cc:2369] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 12:39:26.390887 11827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } attempt: 5
I20260811 12:39:26.394233 11912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [term 1 FOLLOWER]: Refusing update from remote peer 2612191edaaf4ba1aa9d3010af7e57f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:39:26.407677 11991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:39:26.414366 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.380385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.3.204.125:37111)
0811 12:39:26.380777 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.380782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.389162 (+ 8380us) server_negotiation.cc:407] Connection header received
0811 12:39:26.389604 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.389611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.389704 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.389781 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.393807 (+ 4026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.395020 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.398589 (+ 3569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.399266 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.399374 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.405558 (+ 6184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.405566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.405572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.405635 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.409693 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.411179 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.411186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.411189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.411380 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.412060 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.412065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.412068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.412492 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.412766 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.413723 (+ 957us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.413992 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260811 12:39:26.429868 11968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2612191edaaf4ba1aa9d3010af7e57f7. Latest consensus state: current_term: 1 leader_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.430224 11968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:26.434147 12023 mvcc.cc:205] Tried to move back new op lower bound from 7317307254336344064 to 7317307254156050432. Current Snapshot: MvccSnapshot[applied={T|T < 7317307254336344064}]
Aug 11 12:39:26 dist-test-slave-lgr5 krb5kdc[11579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786451965, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.204.125@KRBTEST.COM for kudu/127.3.204.124@KRBTEST.COM
I20260811 12:39:26.437124 12025 mvcc.cc:205] Tried to move back new op lower bound from 7317307254336344064 to 7317307254156050432. Current Snapshot: MvccSnapshot[applied={T|T < 7317307254336344064}]
I20260811 12:39:26.447606 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.380594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.3.204.124:41839)
0811 12:39:26.380760 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.380766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.408416 (+ 27650us) server_negotiation.cc:407] Connection header received
0811 12:39:26.408925 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.408932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.409042 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.409122 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.414346 (+ 5224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.415445 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.417388 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.418161 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.418302 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.436923 (+ 18621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.436930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.436935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.436989 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.440567 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.444488 (+ 3921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.444494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.444498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.444658 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.445113 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.445118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.445122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.445520 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.445780 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.446329 (+ 549us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.446840 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 12:39:26.447778 11968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.448052 11968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:26.452028 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.379249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.3.204.126:46287)
0811 12:39:26.380098 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.380106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.411265 (+ 31159us) server_negotiation.cc:407] Connection header received
0811 12:39:26.411880 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.411886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.411990 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.412060 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.420535 (+ 8475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.421682 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.423921 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.424618 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.424747 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.440040 (+ 15293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.440048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.440052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.440136 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.443733 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.446642 (+ 2909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.446649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.446652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.446824 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.447397 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.447402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.447405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.447812 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.448070 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.448666 (+ 596us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.448904 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673}
I20260811 12:39:26.452452 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.391578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.204.124:41839 (local address 127.0.0.1:36140)
0811 12:39:26.392430 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0811 12:39:26.392453 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 12:39:26.392570 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 12:39:26.418718 (+ 26148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 12:39:26.418724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 12:39:26.419210 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0811 12:39:26.419406 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.419418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.426127 (+ 6709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.426134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.427592 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 12:39:26.427614 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.430779 (+ 3165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.430786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 12:39:26.431032 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.432458 (+ 1426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 12:39:26.432485 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 12:39:26.435699 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 12:39:26.443695 (+ 7996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.443701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.443704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.444315 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.449024 (+ 4709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 12:39:26.449031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 12:39:26.449035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 12:39:26.449245 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 12:39:26.451099 (+ 1854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 12:39:26.451104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 12:39:26.451276 (+ 172us) client_negotiation.cc:769] Sending connection context
0811 12:39:26.451436 (+ 160us) client_negotiation.cc:240] Negotiation successful
0811 12:39:26.451767 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722}
I20260811 12:39:26.453399 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.393069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.3.204.124:41839)
0811 12:39:26.418370 (+ 25301us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.418376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.418402 (+ 26us) server_negotiation.cc:407] Connection header received
0811 12:39:26.418463 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.418468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.418550 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.418673 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.419876 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.425923 (+ 6047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.427812 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.428539 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.435576 (+ 7037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.436180 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 12:39:26.436210 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 12:39:26.436216 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 12:39:26.436257 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 12:39:26.443514 (+ 7257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.448479 (+ 4965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.448485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.448489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.448654 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 12:39:26.450526 (+ 1872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 12:39:26.450532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 12:39:26.450535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 12:39:26.450892 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 12:39:26.451074 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.452766 (+ 1692us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.453015 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25142,"thread_start_us":13730,"threads_started":1}
I20260811 12:39:26.454898 12015 catalog_manager.cc:1347] Generated new cluster ID: 03757683dbf248ca93528a27a375ac99
I20260811 12:39:26.455020 12015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 12:39:26.455437 12004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.455679 12004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:26.459189 11827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9: Bootstrap starting.
I20260811 12:39:26.463424 11827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9: Neither blocks nor log segments found. Creating new log.
I20260811 12:39:26.465391 11827 log.cc:824] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9: Log is configured to *not* fsync() on all Append() calls
I20260811 12:39:26.484990 11827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9: No bootstrap required, opened a new log
I20260811 12:39:26.505339 12015 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 12:39:26.507454 12015 catalog_manager.cc:1504] Loading token signing keys...
I20260811 12:39:26.507030 11827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } }
I20260811 12:39:26.507786 11827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252ea753a6324e0d888cd8b3208eede9, State: Initialized, Role: FOLLOWER
I20260811 12:39:26.508728 11827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } }
I20260811 12:39:26.521243 11795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 12:39:26.525586 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "252ea753a6324e0d888cd8b3208eede9"
I20260811 12:39:26.526614 11795 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [term 1 FOLLOWER]: Refusing update from remote peer 2612191edaaf4ba1aa9d3010af7e57f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 12:39:26.530447 11796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2612191edaaf4ba1aa9d3010af7e57f7 in term 1.
I20260811 12:39:26.531716 12004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 12:39:26.534137 12015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7: Generated new TSK 0
I20260811 12:39:26.535331 12015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 12:39:26.542412 11827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 12:39:26.580982 12034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.581532 12034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:26.582129 12039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.582183 12030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2612191edaaf4ba1aa9d3010af7e57f7. Latest consensus state: current_term: 1 leader_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.582666 12030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:26.582799 12039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 12:39:26.604429 12063 catalog_manager.cc:1259] Loaded cluster ID: 03757683dbf248ca93528a27a375ac99
I20260811 12:39:26.604621 12063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9: loading cluster ID for follower catalog manager: success
I20260811 12:39:26.608441 12063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9: acquiring CA information for follower catalog manager: success
I20260811 12:39:26.627074 12063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 252ea753a6324e0d888cd8b3208eede9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 12:39:26.650832 3889 mini_kdc.cc:348] Time spent kdestroy: real 0.100s user 0.000s sys 0.031s
W20260811 12:39:26.706950 11837 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:26.686675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.3.204.125:37111)
0811 12:39:26.686889 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.686896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.704112 (+ 17216us) server_negotiation.cc:407] Connection header received
0811 12:39:26.705322 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.705329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.705453 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.705526 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.706686 (+ 1160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46476: BlockingRecv error: recv got EOF from 127.0.0.1:46476 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260811 12:39:26.709136 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:26.688475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.3.204.124:41839)
0811 12:39:26.688668 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.688674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.705053 (+ 16379us) server_negotiation.cc:407] Connection header received
0811 12:39:26.707648 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.707654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.707760 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.707828 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.708890 (+ 1062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36152: BlockingRecv error: recv got EOF from 127.0.0.1:36152 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260811 12:39:26.711764 11963 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:26.702318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.3.204.126:46287)
0811 12:39:26.702544 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.702550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.709889 (+ 7339us) server_negotiation.cc:407] Connection header received
0811 12:39:26.710413 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.710419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.710502 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.710567 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.711541 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44344: BlockingRecv error: recv got EOF from 127.0.0.1:44344 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 12:39:26.756346 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.741660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46490 (local address 127.3.204.125:37111)
0811 12:39:26.741864 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.741870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.747924 (+ 6054us) server_negotiation.cc:407] Connection header received
0811 12:39:26.748565 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.748572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.748819 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.748974 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:26.749795 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.750943 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.753070 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.753743 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.753854 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.754691 (+ 837us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:26.755451 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:26.755577 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.755837 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.755948 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260811 12:39:26.804181 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:26.790508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.3.204.124:41839)
0811 12:39:26.790723 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.790731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.801998 (+ 11267us) server_negotiation.cc:407] Connection header received
0811 12:39:26.802396 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.802403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.802552 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.802643 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.803845 (+ 1202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36156: BlockingRecv error: recv got EOF from 127.0.0.1:36156 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 12:39:26.855484 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.840242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.3.204.125:37111)
0811 12:39:26.840490 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.840496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.848121 (+ 7625us) server_negotiation.cc:407] Connection header received
0811 12:39:26.848713 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.848720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.848855 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.848939 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:26.849733 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.850928 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.853044 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.853726 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.853826 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.854505 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:26.854685 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:26.854759 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.855040 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.855138 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260811 12:39:26.903519 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:26.890235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.3.204.124:41839)
0811 12:39:26.890474 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.890482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.901665 (+ 11183us) server_negotiation.cc:407] Connection header received
0811 12:39:26.901972 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.901978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.902100 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.902172 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.903288 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36158: BlockingRecv error: recv got EOF from 127.0.0.1:36158 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 12:39:26.942437 12004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612191edaaf4ba1aa9d3010af7e57f7" member_type: VOTER last_known_addr { host: "127.3.204.126" port: 46287 } } peers { permanent_uuid: "252ea753a6324e0d888cd8b3208eede9" member_type: VOTER last_known_addr { host: "127.3.204.125" port: 37111 } } peers { permanent_uuid: "b8322cc2e767463aa323231637e2ed89" member_type: VOTER last_known_addr { host: "127.3.204.124" port: 41839 } } }
I20260811 12:39:26.942775 12004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2612191edaaf4ba1aa9d3010af7e57f7 [sys.catalog]: This master's current role is: LEADER
I20260811 12:39:26.953263 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.945084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.3.204.125:37111)
0811 12:39:26.945249 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.945255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.945312 (+ 57us) server_negotiation.cc:407] Connection header received
0811 12:39:26.945629 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.945635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.945745 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.945810 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:26.946582 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.947902 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.950636 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.951424 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.951538 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.952253 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:26.952429 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:26.952494 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.952729 (+ 235us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.952847 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260811 12:39:26.971463 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:26.969465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.3.204.124:41839)
0811 12:39:26.969661 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.969667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.969998 (+ 331us) server_negotiation.cc:407] Connection header received
0811 12:39:26.970141 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.970147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.970243 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.970362 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.971136 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36172: BlockingRecv error: recv got EOF from 127.0.0.1:36172 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 12:39:26.984337 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:26.978333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.3.204.125:37111)
0811 12:39:26.978599 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.978605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.978796 (+ 191us) server_negotiation.cc:407] Connection header received
0811 12:39:26.978979 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.978985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.979080 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.979185 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:26.979564 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.980748 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.982293 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:26.982952 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:26.983092 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:26.983457 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:26.983616 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:26.983721 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:26.983890 (+ 169us) server_negotiation.cc:299] Negotiation successful
0811 12:39:26.984022 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260811 12:39:26.993840 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:26.992023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.3.204.124:41839)
0811 12:39:26.992301 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:26.992307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:26.992520 (+ 213us) server_negotiation.cc:407] Connection header received
0811 12:39:26.992691 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:26.992697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:26.992796 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:26.992913 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:26.993607 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36186: BlockingRecv error: recv got EOF from 127.0.0.1:36186 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
I20260811 12:39:27.011570 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.005340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.3.204.125:37111)
0811 12:39:27.005566 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.005573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.005887 (+ 314us) server_negotiation.cc:407] Connection header received
0811 12:39:27.006074 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.006080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.006186 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.006306 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.006729 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.007874 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.009374 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.010056 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.010207 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.010660 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.010847 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.010969 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.011100 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.011200 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260811 12:39:27.019824 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:27.017918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.3.204.124:41839)
0811 12:39:27.018187 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.018194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.018451 (+ 257us) server_negotiation.cc:407] Connection header received
0811 12:39:27.018613 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.018618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.018723 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.018852 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:27.019572 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36200: BlockingRecv error: recv got EOF from 127.0.0.1:36200 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260811 12:39:27.045608 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.039186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.3.204.125:37111)
0811 12:39:27.039404 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.039410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.039728 (+ 318us) server_negotiation.cc:407] Connection header received
0811 12:39:27.039908 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.039914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.040019 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.040170 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.040650 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.041909 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.043457 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.044160 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.044323 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.044772 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.044940 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.045054 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.045211 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.045313 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260811 12:39:27.053792 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:27.051756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.3.204.124:41839)
0811 12:39:27.051942 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.051949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.052379 (+ 430us) server_negotiation.cc:407] Connection header received
0811 12:39:27.052581 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.052585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.052677 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.052766 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:27.053560 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36210: BlockingRecv error: recv got EOF from 127.0.0.1:36210 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 12:39:27.096536 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.090223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.3.204.125:37111)
0811 12:39:27.090470 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.090476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.090888 (+ 412us) server_negotiation.cc:407] Connection header received
0811 12:39:27.091105 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.091110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.091188 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.091277 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.091785 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.092627 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.094577 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.095062 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.095198 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.095697 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.095830 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.095900 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.096057 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.096161 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260811 12:39:27.105170 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:27.103055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.3.204.124:41839)
0811 12:39:27.103221 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.103225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.103597 (+ 372us) server_negotiation.cc:407] Connection header received
0811 12:39:27.103791 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.103795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.103884 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.103970 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:27.104942 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36212: BlockingRecv error: recv got EOF from 127.0.0.1:36212 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 12:39:27.177946 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.172694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.3.204.125:37111)
0811 12:39:27.172924 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.172930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.173261 (+ 331us) server_negotiation.cc:407] Connection header received
0811 12:39:27.173490 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.173495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.173569 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.173650 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.174116 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.174844 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.176256 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.176690 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.176807 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.177218 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.177345 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.177434 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.177571 (+ 137us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.177655 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260811 12:39:27.185618 12024 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 12:39:27.183736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.3.204.124:41839)
0811 12:39:27.183905 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.183910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.184267 (+ 357us) server_negotiation.cc:407] Connection header received
0811 12:39:27.184488 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.184493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.184570 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.184671 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 12:39:27.185439 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36226: BlockingRecv error: recv got EOF from 127.0.0.1:36226 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 12:39:27.319286 11985 catalog_manager.cc:1259] Loaded cluster ID: 03757683dbf248ca93528a27a375ac99
I20260811 12:39:27.319391 11985 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89: loading cluster ID for follower catalog manager: success
I20260811 12:39:27.322660 11985 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89: acquiring CA information for follower catalog manager: success
I20260811 12:39:27.323153 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.317501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.3.204.125:37111)
0811 12:39:27.317712 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.317718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.317981 (+ 263us) server_negotiation.cc:407] Connection header received
0811 12:39:27.318204 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.318208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.318282 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.318359 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.318886 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.319627 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.321520 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.321925 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.322022 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.322471 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.322584 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.322649 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.322829 (+ 180us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.322908 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 12:39:27.324925 11985 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8322cc2e767463aa323231637e2ed89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 12:39:27.334877 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.329171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.3.204.124:41839)
0811 12:39:27.329418 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.329426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.329817 (+ 391us) server_negotiation.cc:407] Connection header received
0811 12:39:27.330024 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.330031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.330172 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.330342 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.330755 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.331529 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.332911 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.333332 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.333427 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.333854 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.334366 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.334489 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.334526 (+ 37us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.334595 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 12:39:27.349519 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.341292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.3.204.125:37111)
0811 12:39:27.341496 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.341504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.342727 (+ 1223us) server_negotiation.cc:407] Connection header received
0811 12:39:27.343020 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.343026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.343141 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.343214 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.344012 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.345165 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.347317 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.347868 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.348002 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.348658 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.348796 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.348886 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.349154 (+ 268us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.349237 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 12:39:27.350131 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.341352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.3.204.124:41839)
0811 12:39:27.341558 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.341564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.342639 (+ 1075us) server_negotiation.cc:407] Connection header received
0811 12:39:27.342954 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.342959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.343053 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.343124 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.344012 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.345142 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.348208 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.348763 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.348874 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.349403 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.349552 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.349614 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.349845 (+ 231us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.349935 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 12:39:27.351642 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 12:39:27.341227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.3.204.126:46287)
0811 12:39:27.341771 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0811 12:39:27.341777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 12:39:27.343309 (+ 1532us) server_negotiation.cc:407] Connection header received
0811 12:39:27.343507 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 12:39:27.343523 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 12:39:27.343785 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 12:39:27.344002 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 12:39:27.346105 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.347179 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.349101 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 12:39:27.349711 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 12:39:27.349824 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 12:39:27.350566 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 12:39:27.351068 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 12:39:27.351172 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 12:39:27.351309 (+ 137us) server_negotiation.cc:299] Negotiation successful
0811 12:39:27.351380 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":306,"threads_started":1}
I20260811 12:39:27.356473 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11603
I20260811 12:39:27.357681 11705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:27.470599 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11603
I20260811 12:39:27.502178 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11730
I20260811 12:39:27.503360 11820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:27.611749 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11730
I20260811 12:39:27.636782 3889 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu pid 11840
I20260811 12:39:27.637898 11944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 12:39:27.745597 3889 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1nKHSZ/build/asan/bin/kudu with pid 11840
2026-08-11T12:39:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4674 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8961 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (382409 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST